Emotion recognition based on brain and machine collaborative intelligence

Dongjun LIU; Yuhan WANG; Wenfen LING; Yong PENG; Wanzeng KONG

doi:10.11959/j.issn.2096-6652.202107

您当前的位置：

首页 >

文章列表页 >

Emotion recognition based on brain and machine collaborative intelligence

Special topic:emotional brain computer interface | 更新时间：2024-06-05

- Emotion recognition based on brain and machine collaborative intelligence
- Chinese Journal of Intelligent Science and Technology Vol. 3, Issue 1, Pages: 65-75(2021)
- 作者机构：
  
  1. 杭州电子科技大学计算机学院，浙江杭州 310018
  2. 浙江省脑机协同智能重点实验室，浙江杭州 310018
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-6652.202107
  CLC： TP18，TN911.7，R318
- Published Online：2021-03，
  
  Published：15 March 2021
- 稿件说明：
移动端阅览
Dongjun LIU, Yuhan WANG, Wenfen LING, et al. Emotion recognition based on brain and machine collaborative intelligence[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 65-75.
DOI：

Dongjun LIU, Yuhan WANG, Wenfen LING, et al. Emotion recognition based on brain and machine collaborative intelligence[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 65-75. DOI： 10.11959/j.issn.2096-6652.202107.

摘要

面部表情识别是一种直接、有效的情绪识别模式。机器学习可以对图像表情进行形式化表征，但由于缺乏大脑的认知表征能力，在小样本数据集或复杂表情（伪装）数据集上的识别性能并不理想。针对此问题，将机器人工智能的形式化表征与人脑通用智能的情感认知能力结合，提出一种基于脑机协同智能的情绪识别方法。首先，从脑电图信号中提取脑电情感特征，以获取大脑对情绪的认知表征。其次，从情感图像中提取图像的视觉特征，以获取机器对情绪的形式化表征。为了增强机器模型的泛化能力，在特征学习中引入样本间的迁移适配。在得到图像视觉特征和脑电情感特征后，采用随机森林回归模型训练得到图像视觉特征与脑电情感特征之间的脑机映射关系。测试图像的图像视觉特征经过脑机映射关系产生虚拟脑电情感特征，然后将虚拟脑电情感特征与图像视觉特征进行融合，用于情绪识别。该方法已经在中国面孔表情图片系统上进行了验证，发现对7种情绪的平均识别准确率为88.51%，相比单纯基于图像的方法，提升了3%～5%。

Abstract

Emotion recognition is a direct and effective mode of emotion recognition.Machine learning relies on the formal representation of image expressions

lacks the cognitive representation ability of the brain

and has poor recognition performance on small sample data sets or complex expression (camouflage) data sets.To this end

the formal representation of machine artificial intelligence was combined with the emotional cognitive ability of human brain general intelligence

and a brain-machine collaborative intelligence emotion recognition method was proposed.Firstly

electroencephalogram (EEG) emotional features were extracted from EEG to obtain the brain’s cognitive representation of emotions.Secondly

the visual features of the image were extracted from the emotional image to obtain the machine’s formal representation of the emotion.In order to enhance the generalization ability of the machine model

the transfer adaptation between samples was introduced in the feature learning.After obtaining image visual features and EEG emotional features

the random forest regression model was trained to obtain the brain-machine mapping relationship between image visual features and EEG emotional features.The visual features of the test image were generated through the brain-machine mapping relationship to generate virtual EEG emotional features

and then the virtual EEG emotional features and image visual features were fused for emotion recognition.This method has been verified on the Chinese facial affective picture system (CFAPS) and found that the average recognition accuracy of the seven emotions is 88.51%

which is 3%～5% higher than the image-based method.

关键词

Keywords

references

聂聃 , 王晓韡 , 段若男 , 等 . 基于脑电的情绪识别研究综述 [J ] . 中国生物医学工程学报 , 2012 , 31 ( 4 ): 595 - 606 .

NIE D , WANG X W , DUAN R N , et al . Research review of emotion recognition based on EEG [J ] . Chinese Journal of Biomedical Engineering , 2012 , 31 ( 4 ): 595 - 606 .

DORNAIKA F , RADUCANU B . Efficient facial expression recognition for human robot interaction [C ] // Proceedings of the 9th International Work-Conference on Artificial Neural Networks . Berlin:Springer , 2007 : 700 - 708 .

HARPER R G , WIENS A N , MATARAZZO J D . Nonverbal communication:the state of the art [M ] .[S.l. ] : John Wiley ＆ Sons , 1978 .

尹旺 , 李惠媛 . 深度学习在脑电情感识别方面的应用研究进展 [J ] . 计算机时代 , 2020 ( 8 ): 14 - 17 .

YIN W , LI H Y . The application research progress of deep learning in EEG emotion recognition [J ] . Computer Era , 2020 ( 8 ): 14 - 17 .

陈景霞 , 郑茹 , 张鹏伟 , 等 . 基于数据空间自适应与共空间模式的脑电情感分类 [J ] . 计算机工程 , 2019 , 45 ( 4 ): 296 - 301 .

CHEN J X , ZHENG R , ZHANG P W , et al . EEG sentiment classification based on data space adaptation and co-space mode [J ] . Computer Engineering , 2019 , 45 ( 4 ): 296 - 301 .

NIE D , WANG X W , SHI L C , et al . EEG-based emotion recognition during watching movies [C ] // Proceedings of the 2011 5th International IEEE/EMBS Conference on Neural Engineering . Piscataway:IEEE Press , 2011 : 667 - 670 .

YANG Y , WU Q , QIU M , et al . Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network [C ] // Proceedings of the 2018 International Joint Conference on Neural Networks . Piscataway:IEEE Press , 2018 : 1 - 7 .

LIPTON Z C , BERKOWITZ J , ELKAN C . A critical review of recurrent neural networks for sequence learning [J ] . arXiv preprint , 2015 ,arXiv:1506.00019.

WALECKI R , PAVLOVIC V , SCHULLER B , et al . Deep structured learning for facial action unit intensity estimation [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2017 : 3405 - 3414 .

LI S , DENG W . Deep facial expression recognition:a survey [J ] . IEEE Transactions on Affective Computing , 2018 .

BREUER R , KIMMEL R . A deep learning perspective on the origin of facial expressions [J ] . arXiv preprint , 2017 ,arXiv:1705.01842.

MOLLAHOSSEINI A , CHAN D , MAHOOR M H . Going deeper in facial expression recognition using deep neural networks [C ] // Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision . Piscataway:IEEE Press , 2016 : 1 - 10 .

王云 . 基于 EEG 与计算机视觉的人脸检索闭环系统 [D ] . 杭州:浙江大学 , 2016 .

WANG Y . Face retrieval closed-loop system based on EEG and computer vision [D ] . Hangzhou:Zhejiang University , 2016 .

LUO Y J , HUANG Y X , LI X Y , et al . Effects of emotion on cognitive processing:series of event-related potentials study [J ] . Advances in Psychological Science , 2006 , 14 ( 4 ): 505 .

KHALILI Z , MORADI M H . Emotion recognition system using brain and peripheral signals:using correlation dimension to improve the results of EEG [C ] // Proceedings of the 2009 International Joint Conference on Neural Networks . Piscataway:IEEE Press , 2009 : 1571 - 1575 .

ZOURIDAKIS G , PATIDAR U , PADHYE N S , et al . Spectral power of brain activity associated with emotion—a pilot MEG study [C ] // Proceedings of the 17th International Conference on Biomagnetism Advances in Biomagnetism-Biomag 2010 . Berlin:Springer , 2010 : 354 - 357 .

LIN Y P , WANG C H , JUNG T P , et al . EEG-based emotion recognition in music listening [J ] . IEEE Transactions on Biomedical Engineering , 2010 , 57 ( 7 ): 1798 - 1806 .

LI X , SONG D W , ZHANG P , et al . Emotion recognition from multi-channel EEG data through convolutional recurrent neural network [C ] // 2016 IEEE International Conference on Bioinformatics and Biomedicine . Piscataway:IEEE Press , 2016 : 352 - 359 .

SCHMIDHUBER J , HOCHREITER S . Long short-term memory [J ] . Neural Computation , 1997 , 9 ( 8 ): 1735 - 1780 .

CHO K , VAN MERRIËNBOER B , BAHDANAU D , et al . On the properties of neural machine translation:encoder-decoder approaches [J ] . arXiv preprint , 2014 arXiv:1409.1259.

LONG M , CAO Y , WANG J , et al . Learning transferable features with deep adaptation networks [C ] // Proceedings of the 32nd International Conference on Machine Learning .[S.l.:s.n. ] , 2015 : 97 - 105 .

GRETTON A , BORGWARDT K M , RASCH M J , et al . A kernel two-sample test [J ] . The Journal of Machine Learning Research , 2012 , 13 ( 1 ): 723 - 773 .

HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition [C ] // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2016 : 770 - 778 .

LING W F , KONG W Z , LONG Y F , et al . Facial emotion recognition based on brain and machine collaborative intelligence [C ] // Proceedings of the 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications . Piscataway:IEEE Press , 2019 : 1 - 5 .

Views

1610

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Cerberus: cross-site social bot detection system based on deep learning

Survey on vision-based railway track defect detection

Visual SLAM based on semantic information and geometric constraints in dynamic environment

Lipsynthesis incorporating audio-visual synchronisation

A survey of deep learning-based MRI stroke lesion segmentation methods

Related Author

TANG Jiawei

LIU Yushan

GAO Min

GONG Qingyuan

WANG Xin

CHEN Yang

CHEN Tianyan

HAN Zeming

Related Institution

Research Institute of Intelligent Complex Systems, Fudan University

School of Computer Science, Fudan University

School of Computer and Big Data, Minjiang University

School of Rail Transit, Fujian Chuanzheng Communications College

School of Transportation, Fujian University of Technology

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：35685 Visits Today：160

⁰