Constructing a reliable and stable emotion recognition system is a critical but challenging issue for realizing an intelligent human-machine interaction. In this study, we contribute a novel channel-frequency convolutional neural network (CFCNN), combined with recurrence quantification analysis (RQA), for the robust recognition of electroencephalogram (EEG) signals collected from different emotion states. We employ movie clips as the stimuli to induce happiness, sadness, and fear emotions and simultaneously measure the corresponding EEG signals. Then the entropy measures, obtained from the RQA operation on EEG signals of different frequency bands, are fed into the novel CFCNN. The results indicate that our system can provide a high emotion recognition accuracy of 92.24% and a relatively excellent stability as well as a satisfactory Kappa value of 0.884, rendering our system particularly useful for the emotion recognition task. Meanwhile, we compare the performance of the entropy measures, extracted from each frequency band, in distinguishing the three emotion states. We mainly find that emotional features extracted from the gamma band present a considerably higher classification accuracy of 90.51% and a Kappa value of 0.858, proving the high relation between emotional process and gamma frequency band.
References
1.
2.
M.
Pantic
and L. J. M.
Rothkrantz
, “Automatic analysis of facial expressions: The state of the art
,” IEEE Trans. Pattern Anal. Mach. Intell.
22
(12
), 1424
–1445
(2000
).3.
Z.
Zeng
, M.
Pantic
, G. I.
Roisman
, and T. S.
Huang
, “A survey of affect recognition methods: Audio, visual, and spontaneous expressions
,” IEEE Trans. Pattern Anal. Mach. Intell.
31
(1
), 39
–58
(2009
).4.
K.
Mistry
, L.
Zhang
, S. C.
Neoh
, C. P.
Lim
, and B.
Fielding
, “A micro-GA embedded PSO feature selection approach to intelligent facial emotion recognition
,” IEEE Trans. Cybern.
47
(6
), 1496
–1509
(2017
).5.
M. E.
Ayadi
, M. S.
Kamel
, and F.
Karray
, “Survey on speech emotion recognition: Features, classification schemes, and databases
,” Pattern Recognit.
44
(3
), 572
–587
(2011
).6.
B.
Schuller
, A.
Batliner
, S.
Steidl
, and D.
Seppi
, “Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge
,” Speech Commun.
53
(9–10
), 1062
–1087
(2011
).7.
H. Y.
Meng
and N.
Bianchi-Berthouze
, “Affective state level recognition in naturalistic facial and vocal expressions
,” IEEE Trans. Cybern.
44
(3
), 315
–328
(2014
).8.
H.
Gunes
and M.
Piccardi
, “Automatic temporal segment detection and affect recognition from face and body display
,” IEEE Trans. Syst. Man Cybern. B
39
(1
), 64
–84
(2009
).9.
A.
Kleinsmith
and N.
Bianchi-Berthouze
, “Affective body expression perception and recognition: A survey
,” IEEE Trans. Affect. Comput.
4
(1
), 15
–33
(2013
).10.
Y.
Gao
, N.
Bianchi-Berthouze
, and H.
Meng
, “What does touch tell us about emotions in touchscreen-based gameplay?
,” ACM Trans. Comput. Hum. Interact.
19
(4
), 31
(2012
).11.
H. P.
Martinez
, Y.
Bengio
, and G. N.
Yannakakis
, “Learning deep physiological models of affect
,” IEEE Comput. Intell. Mag.
8
(2
), 20
–33
(2013
).12.
Y. P.
Lin
, C. H.
Wang
, T. P.
Jung
, T. L.
Wu
, S. K.
Jeng
, J. R.
Duann
, and J. H.
Chen
, “EEG based emotion recognition in music listening
,” IEEE Trans. Biomed. Eng.
57
(7
), 1798
–1806
(2010
).13.
P. C.
Petrantonakis
and L. J.
Hadjileontiadis
, “Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis
,” IEEE Trans. Affect. Comput.
1
(2
), 81
–97
(2010
).14.
G.
Chanel
, C.
Rebetez
, M.
Btrancourt
, and T.
Pun
, “Emotion assessment from physiological signals for adaptation of game difficulty
,” IEEE Trans. Syst. Man Cybern. A
41
(6
), 1052
–1063
(2011
).15.
H.
Xu
and K. N.
Plataniotis
, “Affect recognition using EEG signal
,” in Proceedings of 14th IEEE International Workshop on Multimedia Signal Processing (MMSP)
(IEEE
, 2012
), pp. 299
–304
.16.
S. K.
Hadjidimitriou
and L. J.
Hadjileontiadis
, “EEG-based classification of music appraisal responses using time-frequency analysis and familiarity ratings
,” IEEE Trans. Affect. Comput.
4
(2
), 161
–172
(2013
).17.
X.
Chai
, Q. S.
Wang
, Y. P.
Zhao
, Y. Q.
Li
, D.
Liu
, X.
Liu
, and O.
Bai
, “A fast, efficient domain adaptation technique for cross-domain electroencephalography (EEG)-based emotion recognition
,” Sensors
17
(5
), 1014
(2017
).18.
G. L.
Ahern
and G. E.
Schwartz
, “Differential lateralization for positive and negative emotion in the human brain: EEG spectral analysis
,” Neuropsychologia
23
(6
), 745
–755
(1985
).19.
U. R.
Acharya
, O.
Faust
, N.
Kannathal
, T. J.
Chua
, and S.
Laxminarayan
, “Non-linear analysis of EEG signals at various sleep stages
,” Comput. Methods Programs Biomed.
80
(1
), 37
–45
(2005
).20.
K. C.
Chua
, V.
Chandran
, U. R.
Acharya
, and C. M.
Lim
, “Analysis of epileptic EEG signals using higher order spectra
,” J. Med. Eng. Technol.
33
(1
), 42
–50
(2009
).21.
Z. D.
Mu
, J. F.
Hu
, J. L.
Min
, and J. H.
Yin
, “Comparison of different entropies as features for person authentication based on EEG signals
,” IET Biom.
6
(6
), 409
–417
(2017
).22.
O.
De Wel
, M.
Lavanga
, A. C.
Dorado
, K.
Jansen
, A.
Dereymaeker
, G.
Naulaers
, and S.
Van Huffel
, “Complexity analysis of neonatal EEG using multiscale entropy: Applications in brain maturation and sleep stage classification
,” Entropy
19
(10
), 516
(2017
).23.
Z. K.
Gao
, Q.
Cai
, Y. X.
Yang
, W. D.
Dang
, and S. S.
Zhang
, “Multiscale limited penetrable horizontal visibility graph for analyzing nonlinear time series
,” Sci. Rep.
6
, 35662
(2016
).24.
Z. K.
Gao
, Q.
Cai
, Y. X.
Yang
, N.
Dong
, and S. S.
Zhang
, “Visibility graph from adaptive optimal kernel time-frequency representation for classification of epileptic from EEG
,” Int. J. Neural Syst.
27
(4
), 1750005
(2017
).25.
J.-P.
Eckmann
, S. O.
Kamphorst
, and D.
Ruelle
, “Recurrence plots of dynamical systems
,” Europhys. Lett.
4
(9
), 973
–977
(1987
).26.
N.
Marwan
, M. C.
Romano
, M.
Thiel
, and J.
Kurths
, “Recurrence plots for the analysis of complex systems
,” Phys. Rep.
438
(5–6
), 237
–329
(2007
).27.
L. L.
Trulla
, A.
Giuliani
, J. P.
Zbilut
, and C. L.
Webber
, “Recurrence quantification analysis of the logistic equation with transients
,” Phys. Lett. A
223
(4
), 255
–260
(1996
).28.
J. P.
Zbilut
and C. L.
Webber
, Jr., “Embeddings and delays as derived from quantification of recurrence plots
,” Phys. Lett. A
171
(3–4
), 199
–203
(1992
).29.
H.
Yang
, “Multiscale recurrence quantification analysis of spatial cardiac vector cardiogram signals
,” IEEE Trans. Biomed. Eng.
58
(2
), 339
–347
(2011
).30.
X. L.
Li
, G. X.
Ouyang
, X.
Yao
, and X. P.
Guan
, “Dynamical characteristics of pre-epileptic seizures in rats with recurrence quantification analysis
,” Phys. Lett. A
333
(1–2
), 164
–171
(2004
).31.
U. R.
Acharya
, V. S.
Sree
, S.
Chattopadhyay
, W. W.
Yu
, and A. P. C.
Alvin
, “Application of recurrence quantification analysis for the automated identification of epileptic EEG signals
,” Int. J. Neural Syst.
21
(3
), 199
–211
(2011
).32.
L. H.
Song
, D. S.
Lee
, and S. I.
Kim
, “Recurrence quantification analysis of sleep electroencephalogram in sleep apnea syndrome in humans
,” Neurosci. Lett.
366
(2
), 148
–153
(2004
).33.
U. R.
Acharya
, V. K.
Sudarshan
, H.
Adeli
, J.
Santhosh
, J. E. W.
Koh
, S. D.
Puthankatti
, and A.
Adeli
, “A novel depression diagnosis index using nonlinear features in EEG signals
,” Eur. Neurol.
74
(1–2
), 79
–83
(2015
).34.
N.
Talebi
, A. M.
Nasrabadi
, and T.
Curran
, “Investigation of changes in EEG complexity during memory retrieval: The effect of midazolam
,” Cogn. Neurodyn.
6
(6
), 537
–546
(2012
).35.
K.
Becker
, G.
Schneider
, M.
Eder
, A.
Ranft
, E. F.
Kochs
, W.
Zieglgansberger
, and H. U.
Dodt
, “Anaesthesia monitoring by recurrence quantification analysis of EEG data
,” PLoS One
5
(1
), e8876
(2010
).36.
X. W.
Wang
, D.
Nie
, and B. L.
Lu
, “Emotional state classification from EEG data using machine learning approach
,” Neurocomputing
129
, 94
–106
(2014
).37.
H.
Shahabi
and S.
Moghimi
, “Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity
,” Comput. Hum. Behav.
58
, 231
–239
(2016
).38.
G. E.
Hinton
and R. R.
Salakhutdinov
, “Reducing the dimensionality of data with neural networks
,” Science
313
(5786
), 504
–507
(2006
).39.
Y.
LeCun
and Y.
Bengio
, “Convolutional networks for images, speech, and time series
,” in The Handbook of Brain Theory and Neural Networks
(MIT Press
, 1995
), pp. 255
–258
.40.
S.
Hochreiter
and J.
Schmidhuber
, “Long short-term memory
,” Neural Comput.
9
(8
), 1735
–1780
(1997
).41.
F. A.
Gers
, N. N.
Schraudolph
, and J.
Schmidhuber
, “Learning precise timing with LSTM recurrent networks
,” J. Mach. Learn. Res.
3
(1
), 115
–143
(2003
).42.
W. L.
Zheng
and B. L.
Lu
, “Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks
,” IEEE Trans. Auton. Ment. Dev.
7
(3
), 162
–175
(2015
).43.
X.
Li
, D. W.
Song
, P.
Zhang
, G. L.
Yu
, Y. X.
Hou
, and B.
Hu
, “Emotion recognition from multi-channel EEG data through convolutional recurrent neural network
,” in Proceedings of IEEE International Conference on Bioinformatics and Biomedicine (BIBM)
(IEEE COMPUTER SOC
, 2016
), pp. 352
–359
.44.
G. L.
Yu
, X.
Li
, D. W.
Song
, X. Z.
Zhao
, P.
Zhang
, Y. X.
Hou
, and B.
Hu
, “Encoding physiological signals as images for affective state recognition using convolutional neural networks
,” in Proceedings of 38th International Conference on IEEE Engineering Medicine and Biology Society (EMBC)
(IEEE
, 2016
), pp. 812
–815
.45.
T. K.
Marcha
, S. C.
Chapman
, and R. O.
Dendy
, “Recurrence plot statistics and the effect of embedding
,” Physica D
200
, 171
–184
(2005
).46.
B.
Goswami
, N.
Marwan
, G.
Feulner
, and J.
Kurths
, “How do global temperature drivers influence each other? A network perspective using recurrences
,” Eur. Phys. J. Spec. Top.
222
, 861
–873
(2013
).47.
E. J.
Ngamga
, S.
Bialonski
, N.
Marwan
, J.
Kurths
, C.
Geier
, and K.
Lehnertz
, “Evaluation of selected recurrence measures in discriminating pre-ictal and inter-ictal periods from epileptic EEG data
,” Phys. Lett. A
380
(16
), 1419
–1425
(2016
).48.
N.
Marwan
, “How to avoid potential pitfalls in recurrence plot based data analysis
,” Int. J. Bifurcat. Chaos
21
(4
), 1003
–1017
(2011
).49.
S.
Schinkel
, O.
Dimigen
, and N.
Marwan
, “Selection of recurrence threshold for signal detection
,” Eur. Phys. J. Spec. Top.
164
, 45
–53
(2008
).50.
V.
Nair
and G. E.
Hinton
, “Rectified linear units improve restricted Boltzmann machines
,” in Proceedings of 27th International Conference on Machine Learning
(Omnipress
, USA
, 2010
), pp. 807
–814
.51.
F.
Chollet
, see https://github.com/fchollet/keras for “Keras: Deep Learning Library for Theano and TensorFlow
” (2015
).52.
S.
Ioffe
and C.
Szegedy
, “Batch normalization: Accelerating deep network training by reducing internal covariate shift
,” Proceedings of Machine Learning Research (PMLR)
37
, 448
–456
(2015
).53.
X.
Glorot
and Y.
Bengio
, “Understanding the difficulty of training deep feedforward neural networks
,” Proceedings of Machine Learning Research (PMLR)
9
, 249
–256
(2010
).54.
L.
Bottou
, “Large-scale machine learning with stochastic gradient descent
,” in Proceedings of 19th International Conference on Computational Statistics
(Physica-Verlag
, Springer
, 2010
), pp. 177–186.55.
D. E.
Rumelhart
, G. E.
Hinton
, and R. J.
Williams
, “Learning representations by back-propagating errors
,” Nature
323
(6088
), 533
–536
(1986
).56.
J.
Cohen
, “A coefficient of agreement for nominal scales
,” Educ. Psychol. Meas.
20
(1
), 37
–46
(1960
).57.
Y. R.
Tabar
and U.
Halici
, “A novel deep learning approach for classification of EEG motor imagery signals
,” J. Neural Eng.
14
, 016003
(2017
).58.
S.
Aydin
, S.
Demirtas
, K.
Ates
, and M. A.
Tunga
, “Emotion recognition with eigen features of frequency band activities embedded in induced brain oscillations mediated by affective pictures
,” Int. J. Neural Syst.
26
(3
), 1650013
(2016
).59.
M. M.
Muller
, A.
Keil
, T.
Gruber
, and T.
Elbert
, “Processing of affective pictures modulates right hemispheric gamma band EEG activity
,” Clin. Neurophysiol.
110
(11
), 1913
–1920
(1999
).60.
M. A.
Kisley
and Z. M.
Cornwell
, “Gamma and beta neural activity evoked during a sensory gating paradigm: Effects of auditory, somatosensory and cross-modal stimulation
,” Clin. Neurophysiol.
117
(11
), 2549
–2563
(2006
).© 2018 Author(s).
2018
Author(s)
You do not currently have access to this content.
