基于脑机协同智能的情绪识别

刘栋军; 王宇涵; 凌文芬; 彭勇; 孔万增

doi:10.11959/j.issn.2096-6652.202107

您当前的位置：

首页 >

文章列表页 >

基于脑机协同智能的情绪识别

专题：情感脑机接口 | 更新时间：2024-06-05

- 基于脑机协同智能的情绪识别
- Emotion recognition based on brain and machine collaborative intelligence
- 智能科学与技术学报 2021年3卷第1期页码：65-75
- 作者机构：
  
  1. 杭州电子科技大学计算机学院，浙江杭州 310018
  2. 浙江省脑机协同智能重点实验室，浙江杭州 310018
- 作者简介：
  
  [ "刘栋军（1997- ），男，杭州电子科技大学计算机学院硕士生，主要研究方向为脑机协同智能、情绪识别。" ]
  [ "王宇涵（1997- ），女，杭州电子科技大学计算机学院硕士生，主要研究方向为脑机协同智能、情绪识别。" ]
  [ "凌文芬（1995- ），女，杭州电子科技大学计算机学院硕士生，主要研究方向为情绪生理信号、特征提取、情绪识别。" ]
  [ "彭勇（1985- ），男，博士，杭州电子科技大学认知与智能计算研究所副教授，主要研究方向为机器学习、模式识别与脑机交互算法及应用。" ]
  [ "孔万增（1980- ），男，博士，杭州电子科技大学计算机学院认知与智能计算研究所教授、博士生导师，杭州电子科技大学研究生院副院长、浙江省脑机协同智能重点实验室主任，主要研究方向为人工智能与模式识别、嵌入式可穿戴计算、脑机交互与认知计算等。" ]
- 基金信息：
  
  国家重点研发计划基金资助项目;The National Key Research and Development Program of China(2017YFE0116800);国家自然科学基金资助项目;The National Natural Science Foundation of China(U20B2074;U1909202;61971173);浙江省科技计划项目;Science and Technology Program of Zhejiang Province(2018C04012);浙江省“脑机协同智能”重点实验室开放基金项目;Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province(20200E10010)
- DOI：10.11959/j.issn.2096-6652.202107
  中图分类号： TP18，TN911.7，R318
- 网络出版日期：2021-03，
  
  纸质出版日期：2021-03-15
- 稿件说明：
移动端阅览
刘栋军, 王宇涵, 凌文芬, 等. 基于脑机协同智能的情绪识别[J]. 智能科学与技术学报, 2021,3(1):65-75.

Dongjun LIU, Yuhan WANG, Wenfen LING, et al. Emotion recognition based on brain and machine collaborative intelligence[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 65-75.
刘栋军, 王宇涵, 凌文芬, 等. 基于脑机协同智能的情绪识别[J]. 智能科学与技术学报, 2021,3(1):65-75. DOI： 10.11959/j.issn.2096-6652.202107.

Dongjun LIU, Yuhan WANG, Wenfen LING, et al. Emotion recognition based on brain and machine collaborative intelligence[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 65-75. DOI： 10.11959/j.issn.2096-6652.202107.

摘要

面部表情识别是一种直接、有效的情绪识别模式。机器学习可以对图像表情进行形式化表征，但由于缺乏大脑的认知表征能力，在小样本数据集或复杂表情（伪装）数据集上的识别性能并不理想。针对此问题，将机器人工智能的形式化表征与人脑通用智能的情感认知能力结合，提出一种基于脑机协同智能的情绪识别方法。首先，从脑电图信号中提取脑电情感特征，以获取大脑对情绪的认知表征。其次，从情感图像中提取图像的视觉特征，以获取机器对情绪的形式化表征。为了增强机器模型的泛化能力，在特征学习中引入样本间的迁移适配。在得到图像视觉特征和脑电情感特征后，采用随机森林回归模型训练得到图像视觉特征与脑电情感特征之间的脑机映射关系。测试图像的图像视觉特征经过脑机映射关系产生虚拟脑电情感特征，然后将虚拟脑电情感特征与图像视觉特征进行融合，用于情绪识别。该方法已经在中国面孔表情图片系统上进行了验证，发现对7种情绪的平均识别准确率为88.51%，相比单纯基于图像的方法，提升了3%～5%。

Abstract

Emotion recognition is a direct and effective mode of emotion recognition.Machine learning relies on the formal representation of image expressions

lacks the cognitive representation ability of the brain

and has poor recognition performance on small sample data sets or complex expression (camouflage) data sets.To this end

the formal representation of machine artificial intelligence was combined with the emotional cognitive ability of human brain general intelligence

and a brain-machine collaborative intelligence emotion recognition method was proposed.Firstly

electroencephalogram (EEG) emotional features were extracted from EEG to obtain the brain’s cognitive representation of emotions.Secondly

the visual features of the image were extracted from the emotional image to obtain the machine’s formal representation of the emotion.In order to enhance the generalization ability of the machine model

the transfer adaptation between samples was introduced in the feature learning.After obtaining image visual features and EEG emotional features

the random forest regression model was trained to obtain the brain-machine mapping relationship between image visual features and EEG emotional features.The visual features of the test image were generated through the brain-machine mapping relationship to generate virtual EEG emotional features

and then the virtual EEG emotional features and image visual features were fused for emotion recognition.This method has been verified on the Chinese facial affective picture system (CFAPS) and found that the average recognition accuracy of the seven emotions is 88.51%

which is 3%～5% higher than the image-based method.

关键词

Keywords

references

聂聃 , 王晓韡 , 段若男 , 等 . 基于脑电的情绪识别研究综述 [J ] . 中国生物医学工程学报 , 2012 , 31 ( 4 ): 595 - 606 .

NIE D , WANG X W , DUAN R N , et al . Research review of emotion recognition based on EEG [J ] . Chinese Journal of Biomedical Engineering , 2012 , 31 ( 4 ): 595 - 606 .

DORNAIKA F , RADUCANU B . Efficient facial expression recognition for human robot interaction [C ] // Proceedings of the 9th International Work-Conference on Artificial Neural Networks . Berlin:Springer , 2007 : 700 - 708 .

HARPER R G , WIENS A N , MATARAZZO J D . Nonverbal communication:the state of the art [M ] .[S.l. ] : John Wiley ＆ Sons , 1978 .

尹旺 , 李惠媛 . 深度学习在脑电情感识别方面的应用研究进展 [J ] . 计算机时代 , 2020 ( 8 ): 14 - 17 .

YIN W , LI H Y . The application research progress of deep learning in EEG emotion recognition [J ] . Computer Era , 2020 ( 8 ): 14 - 17 .

陈景霞 , 郑茹 , 张鹏伟 , 等 . 基于数据空间自适应与共空间模式的脑电情感分类 [J ] . 计算机工程 , 2019 , 45 ( 4 ): 296 - 301 .

CHEN J X , ZHENG R , ZHANG P W , et al . EEG sentiment classification based on data space adaptation and co-space mode [J ] . Computer Engineering , 2019 , 45 ( 4 ): 296 - 301 .

NIE D , WANG X W , SHI L C , et al . EEG-based emotion recognition during watching movies [C ] // Proceedings of the 2011 5th International IEEE/EMBS Conference on Neural Engineering . Piscataway:IEEE Press , 2011 : 667 - 670 .

YANG Y , WU Q , QIU M , et al . Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network [C ] // Proceedings of the 2018 International Joint Conference on Neural Networks . Piscataway:IEEE Press , 2018 : 1 - 7 .

LIPTON Z C , BERKOWITZ J , ELKAN C . A critical review of recurrent neural networks for sequence learning [J ] . arXiv preprint , 2015 ,arXiv:1506.00019.

WALECKI R , PAVLOVIC V , SCHULLER B , et al . Deep structured learning for facial action unit intensity estimation [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2017 : 3405 - 3414 .

LI S , DENG W . Deep facial expression recognition:a survey [J ] . IEEE Transactions on Affective Computing , 2018 .

BREUER R , KIMMEL R . A deep learning perspective on the origin of facial expressions [J ] . arXiv preprint , 2017 ,arXiv:1705.01842.

MOLLAHOSSEINI A , CHAN D , MAHOOR M H . Going deeper in facial expression recognition using deep neural networks [C ] // Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision . Piscataway:IEEE Press , 2016 : 1 - 10 .

王云 . 基于 EEG 与计算机视觉的人脸检索闭环系统 [D ] . 杭州:浙江大学 , 2016 .

WANG Y . Face retrieval closed-loop system based on EEG and computer vision [D ] . Hangzhou:Zhejiang University , 2016 .

LUO Y J , HUANG Y X , LI X Y , et al . Effects of emotion on cognitive processing:series of event-related potentials study [J ] . Advances in Psychological Science , 2006 , 14 ( 4 ): 505 .

KHALILI Z , MORADI M H . Emotion recognition system using brain and peripheral signals:using correlation dimension to improve the results of EEG [C ] // Proceedings of the 2009 International Joint Conference on Neural Networks . Piscataway:IEEE Press , 2009 : 1571 - 1575 .

ZOURIDAKIS G , PATIDAR U , PADHYE N S , et al . Spectral power of brain activity associated with emotion—a pilot MEG study [C ] // Proceedings of the 17th International Conference on Biomagnetism Advances in Biomagnetism-Biomag 2010 . Berlin:Springer , 2010 : 354 - 357 .

LIN Y P , WANG C H , JUNG T P , et al . EEG-based emotion recognition in music listening [J ] . IEEE Transactions on Biomedical Engineering , 2010 , 57 ( 7 ): 1798 - 1806 .

LI X , SONG D W , ZHANG P , et al . Emotion recognition from multi-channel EEG data through convolutional recurrent neural network [C ] // 2016 IEEE International Conference on Bioinformatics and Biomedicine . Piscataway:IEEE Press , 2016 : 352 - 359 .

SCHMIDHUBER J , HOCHREITER S . Long short-term memory [J ] . Neural Computation , 1997 , 9 ( 8 ): 1735 - 1780 .

CHO K , VAN MERRIËNBOER B , BAHDANAU D , et al . On the properties of neural machine translation:encoder-decoder approaches [J ] . arXiv preprint , 2014 arXiv:1409.1259.

LONG M , CAO Y , WANG J , et al . Learning transferable features with deep adaptation networks [C ] // Proceedings of the 32nd International Conference on Machine Learning .[S.l.:s.n. ] , 2015 : 97 - 105 .

GRETTON A , BORGWARDT K M , RASCH M J , et al . A kernel two-sample test [J ] . The Journal of Machine Learning Research , 2012 , 13 ( 1 ): 723 - 773 .

HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition [C ] // Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2016 : 770 - 778 .

LING W F , KONG W Z , LONG Y F , et al . Facial emotion recognition based on brain and machine collaborative intelligence [C ] // Proceedings of the 2019 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications . Piscataway:IEEE Press , 2019 : 1 - 5 .

浏览量

1610

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

Cerberus：基于深度学习的跨网站社交机器人检测系统

基于视觉的列车轨道缺陷检测综述

动态环境下基于语义信息与几何约束的视觉SLAM系统

融合音画同步的唇形合成研究

基于深度学习的MRI脑卒中病灶分割方法综述