Research on human mental disorders has started exploring the relationship between human facial expression recognition (FER) and symptoms of mental disorders because cases in FER are similar to symptoms in psychiatric disease. With the tremendous success of machine learning and deep learning, various models and architecture of these techniques are being leveraged to achieve better performance. The purpose of this work is to undertake a review of the most recent work on automatic facial emotion recognition (FER) using deep learning. We emphasize these contributions from the deep learning architecture and facial databases and the researchers’ progress by comparing the recommended technique with the findings produced. The study found that facial expression recognition based on cognitive internet of things (CIoT) in psychiatry disease did not yet exist, so it became a big opportunity to develop this research in the future. It can be concluded that this research can assist scholars by examining previous works and providing insights to improve this field and can be used as recommendations for psychiatric disease.

1.
B.
Bediou
,
J.
Brunelin
,
T.
d’Amato
,
S.
Fecteau
,
M.
Saoud
,
M. A.
Hénaff
, and
P.
Krolak Salmon
,
Frontiers in Psychology
3
,
1
(
2012
)
2.
R. J.
Davidson
,
K. M.
Putnam
, and
C. L.
Larson
,
Science
289
,
591
(
2000
)
3.
Dalgleish
,
Nature Reviews Neuroscience
5
,
583
(
2004
)
4.
M.
Cearns
,
T.
Hahn
, and
B. T.
Baune
,
Translational Psychiatry
9
, (
2019
)
5.
P.
Ekman
and
W.
Friesen
, in (
1978
)
6.
E. A.
Clark
,
J.
Kessinger
,
S. E.
Duncan
,
M. Ann
Bell
,
J.
Lahne
,
D. L.
Gallagher
, and
S. F.
O’Keefe
,
Frontiers in Psychology
11
, (
2020
)
7.
M.
Pantic
and
L. J. M.
Rothkrantz
,
IEEE Transactions on Pattern Analysis and Machine Intelligence
22
,
1424
(
2000
)
8.
B.
Fasel
and
J.
Luettin
,
Pattern Recognition
36
,
259
(
2003
)
9.
A. M.
Kring
and
D. M.
Sloan
,
Psychological Assessment
19
,
210
(
2007
)
10.
S. Z.
Li
and
Anil. K.
Jain
, Handbook of Face Recognition,
Second Edi
(
Springer London
,
London
,
2011
)
11.
R.
Gross
,
I.
Matthews
,
J.
Cohn
,
T.
Kanade
, and
S.
Baker
, in
2008 8th IEEE International Conference on Automatic Face & Gesture Recognition
, pp.
1
8
(
2008
)
12.
A.
Mollahosseini
,
B.
Hasani
, and
M. H.
Mahoor
,
IEEE Transactions on Affective Computing
10
,
18
(
2019
)
13.
I. J.
Goodfellow
,
D.
Erhan
,
P. Luc
Carrier
,
A.
Courville
,
M.
Mirza
,
B.
Hamner
,
W.
Cukierski
,
Y.
Tang
,
D.
Thaler
,
D.-H.
Lee
,
Y.
Zhou
,
C.
Ramaiah
,
F.
Feng
,
R.
Li
,
X.
Wang
,
D.
Athanasakis
,
J.
Shawe-Taylor
,
M.
Milakov
,
J.
Park
,
R.
Ionescu
,
M.
Popescu
,
C.
Grozea
,
J.
Bergstra
,
J.
Xie
,
L.
Romaszko
,
B.
Xu
,
Z.
Chuang
, and
Y.
Bengio
,
Neural Networks
64
,
59
(
2015
)
14.
S.
Li
,
W.
Deng
, and
J.
Du
, in
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
, pp.
2584
2593
(
2017
)
15.
O.
Langner
,
R.
Dotsch
,
G.
Bijlstra
,
D. H. J.
Wigboldus
,
S. T.
Hawk
, and
A.
van Knippenberg
,
Cognition and Emotion
24
,
1377
(
2010
)
16.
M.
Pantic
,
M.
Valstar
,
R.
Rademaker
, and
L.
Maat
, in
2005 IEEE International Conference on Multimedia and Expo
, pp.
5
(
2005
)
17.
G.
Zhao
,
X.
Huang
,
M.
Taini
,
S. Z.
Li
, and
M.
Pietikäinen
,
Image and Vision Computing
29
,
607
(
2011
)
18.
L.
Yin
,
X.
Wei
,
Y.
Sun
,
J.
Wang
, and
M. J.
Rosato
, in
7th International Conference on Automatic Face and Gesture Recognition (FGR06)
, pp.
211
216
(
2006
)
19.
A.
Dhall
,
R.
Goecke
,
S.
Lucey
, and
T.
Gedeon
, in
2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops)
, pp.
2106
2112
(
2011
)
20.
P.
Lucey
,
J. F.
Cohn
,
T.
Kanade
,
J.
Saragih
,
Z.
Ambadar
, and
I.
Matthews
, in
2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops
, pp.
94
101
(
2010
)
21.
M. F.
Valstar
,
B.
Jiang
,
M.
Mehu
,
M.
Pantic
, and
K.
Scherer
, in
2011 IEEE International Conference on Automatic Face Gesture Recognition (FG)
, pp.
921
926
(
2011
)
22.
W.-J.
Yan
,
X.
Li
,
S.-J.
Wang
,
G.
Zhao
,
Y.-J.
Liu
,
Y.-H.
Chen
, and
X.
Fu
,
PLoS ONE
9
,
e86041
(
2014
)
23.
M.
Lyons
,
M.
Kamachi
, and
J.
Gyoba
, (
1998
)
24.
AT&T Labs Research
, https://Cam-Orl.Co.Uk/Facedatabase.Html (
1994
)
25.
C. E.
Thomaz
, (
2006
)
26.
G. B.
Huang
,
M.
Mattar
,
T.
Berg
,
E.
Learned-Miller
, and
E.
Learned-Miller
, in
Workshop on Faces in “Real-Life” Images: Detection, Alignment, and Recognition
(
2008
)
27.
D.
Canedo
and
A. J. R.
Neves
,
Applied Sciences (Switzerland)
9
, (
2019
)
28.
S.
Li
and
W.
Deng
,
IEEE Transactions on Affective Computing
1
(
2020
)
29.
W.
Mellouk
and
W.
Handouzi
,
Procedia Computer Science
175
,
689
(
2020
)
30.
M.
Liu
,
S.
Li
,
S.
Shan
,
R.
Wang
, and
X.
Chen
, in Lecture Notes in Computer Science, pp.
143
157
(
Springer
,
2015
)
31.
Z.
Liu
,
P.
Luo
,
X.
Wang
, and
X.
Tang
, in
2015 IEEE International Conference on Computer Vision (ICCV)
, pp.
3730
3738
(
IEEE
,
2015
)
32.
L.
Ale
,
X.
Fang
,
D.
Chen
,
Y.
Wang
, and
N.
Zhang
,
Proceedings - 2019 18th IEEE International Conference on Trust, Security and Privacy in Computing and Communications/13th IEEE International Conference on Big Data Science and Engineering, TrustCom/BigDataSE 2019-Janua
,
707
(
2019
)
33.
X.
Wu
,
D.
Sahoo
, and
S. C. H.
Hoi
,
Neurocomputing
396
,
39
(
2020
)
34.
M. S.
Hossain
and
G.
Muhammad
,
Information Fusion
49
,
69
(
2019
)
35.
A.
Mollahosseini
,
D.
Chan
, and
M. H.
Mahoor
, in
2016 IEEE Winter Conference on Applications of Computer Vision (WACV)
, pp.
1
10
(
2016
)
36.
A. T.
Lopes
,
E.
de Aguiar
,
A. F.
de Souza
, and
T.
Oliveira-Santos
,
Pattern Recognition
61
,
610
(
2017
)
37.
M.
Mohammadpour
,
H.
Khaliliardali
,
S.
Mohammad
.
R.
Hashemi
, and
Mohammad. M.
AlyanNezhadi
, in
2017 IEEE 4th International Conference on Knowledge-Based Engineering and Innovation (KBEI)
, pp.
0017
0021
(
IEEE
,
2017
)
38.
Y.
Li
,
J.
Zeng
,
S.
Shan
, and
X.
Chen
,
IEEE Transactions on Image Processing
28
,
2439
(
2019
)
39.
J.
Cai
,
O.
Chang
,
X.
Tang
,
C.
Xue
, and
C.
Wei
, in
2018 37th Chinese Control Conference (CCC)
, pp.
9608
9613
(
2018
)
40.
G.
Yolcu
,
I.
Oztel
,
S.
Kazan
,
C.
Oz
,
K.
Palaniappan
,
T. E.
Lever
, and
F.
Bunyak
,
Multimedia Tools and Applications
78
,
31581
(
2019
)
41.
D. K.
Jain
,
P.
Shamsolmoali
, and
P.
Sehdev
,
Pattern Recognition Letters
120
,
69
(
2019
)
42.
D. H.
Kim
,
W. J.
Baddar
,
J.
Jang
, and
Y. M.
Ro
,
IEEE Transactions on Affective Computing
10
,
223
(
2019
)
43.
A.
Agrawal
and
N.
Mittal
,
The Visual Computer
36
,
405
(
2020
)
44.
M.
Shi
,
L.
Xu
, and
X.
Chen
,
IEEE Access
8
,
57606
(
2020
)
45.
M.
Masud
,
G.
Muhammad
,
H.
Alhumyani
,
S. S.
Alshamrani
,
O.
Cheikhrouhou
,
S.
Ibrahim
, and
M. S.
Hossain
,
Computer Communications
152
,
215
(
2020
)
This content is only available via PDF.
You do not currently have access to this content.