基于3D分层卷积融合的多模态生理信号情绪识别

凌文芬; 陈思含; 彭勇; 孔万增

doi:10.11959/j.issn.2096-6652.202108

您当前的位置：

首页 >

文章列表页 >

基于3D分层卷积融合的多模态生理信号情绪识别

专题：情感脑机接口 | 更新时间：2024-06-05

- 基于3D分层卷积融合的多模态生理信号情绪识别
- Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion
- 智能科学与技术学报 2021年3卷第1期页码：76-84
- 作者机构：
  
  1. 杭州电子科技大学计算机学院，浙江杭州 310018
  2. 浙江省脑机协同智能重点实验室，浙江杭州 310018
- 作者简介：
  
  [ "凌文芬（1995- ），女，杭州电子科技大学计算机学院硕士生，主要研究方向为情绪生理信号、特征提取、情绪识别。" ]
  [ "陈思含（1997- ），女，杭州电子科技大学计算机学院硕士生，主要研究方向为情绪生理信号、特征提取、情绪识别。" ]
  [ "彭勇（1985- ），男，博士，杭州电子科技大学计算机学院副教授，主要研究方向为机器学习、模式识别与脑机交互算法及应用。" ]
  [ "孔万增（1980- ），男，博士，杭州电子科技大学计算机学院认知与智能计算研究所教授、博士生导师，杭州电子科技大学研究生院副院长，浙江省脑机协同智能重点实验室主任，主要研究方向为人工智能与模式识别、嵌入式可穿戴计算、脑机交互与认知计算等。" ]
- 基金信息：
  
  国家重点研发计划基金资助项目;The National Key Research and Development Program of China(2017YFE0116800);国家自然科学基金资助项目;The National Natural Science Foundation of China(U1909202;U20B2074;61971173);浙江省科技计划项目;Science and Technology Program of Zhejiang Province(2018C04012);浙江省“脑机协同智能”重点实验室开放基金项目;Key Laboratory of Brain Machine Collaborative Intelligence of Zhejiang Province(20200E10010)
- DOI：10.11959/j.issn.2096-6652.202108
  中图分类号： TP18，TN911.7，R318
- 网络出版日期：2021-03，
  
  纸质出版日期：2021-03-15
- 稿件说明：
移动端阅览
凌文芬, 陈思含, 彭勇, 等. 基于3D分层卷积融合的多模态生理信号情绪识别[J]. 智能科学与技术学报, 2021,3(1):76-84.

Wenfen LING, Sihan CHEN, Yong PENG, et al. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 76-84.
凌文芬, 陈思含, 彭勇, 等. 基于3D分层卷积融合的多模态生理信号情绪识别[J]. 智能科学与技术学报, 2021,3(1):76-84. DOI： 10.11959/j.issn.2096-6652.202108.

Wenfen LING, Sihan CHEN, Yong PENG, et al. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 76-84. DOI： 10.11959/j.issn.2096-6652.202108.

摘要

近年来，脑电等生理信号由于能客观体现真实情绪已逐渐成为情绪识别研究的热门对象。然而，单模态的脑电信号存在情绪信息特征不完备问题，多模态生理信号存在情绪信息交互不充分问题。针对这些问题，提出基于3D分层卷积的多模态特征融合模型，旨在充分挖掘多模态交互关系，更准确地刻画情感信息。首先分别通过深度可分离卷积网络提取脑电、眼电和肌电3种模态的生理信号的多模态初级情绪特征信息，再对得到的多模态初级情绪特征信息进行3D卷积融合操作，实现两两模态间的局部交互以及所有模态间的全局交互，获取包含不同生理信号情绪特征的多模态融合特征。实验结果表明，提出的模型在DEAP数据集的效价、唤醒度的二分类和四分类任务中达到了98%的平均准确率。

Abstract

In recent years

physiological signals such as electroencephalograhpy (EEG) have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However

the single-modal EEG signal has the problem of incomplete emotional information representation

and the multi-modal physiological signal has the problem of insufficient emotional information interaction.Therefore

a 3D hierarchical convolutional fusion model was proposed

which aimed to fully explore multi-modal interaction relationships and more accurately describe emotional information.The method first extracted the primary emotional representation information of EEG

electro-oculogram (EOG) and electromyography (EMG) by depthwise separable convolution network

and then performed 3D convolution fusion operation on the obtained multi-modal primary emotional representation information to realize the pairwise mode local interactions between states and global interactions among all modalities

so as to obtain multi-modal fusion representations containing emotional characteristics of different physiological signals.The results show that the accuracy in the valence and arousal of the two-class and four-class tasks on DEAP dataset are both 98% by the proposed model.

关键词

Keywords

references

DORNAIKA F , RADUCANU B . Efficient facial expression recognition for human robot interaction [C ] // Proceedings of the 9th International Work-Conference on Artificial Neural Networks . Berlin:Springer , 2007 : 700 - 708 .

BARTNECK C , LYONS M J . HCI and the face:towards an art of the soluble [C ] // Proceedings of the 21th International Conference on Human-Computer Interaction . Berlin:Springer , 2007 : 20 - 29 .

DE NADAI S , D'INCÀ M , PARODI F , et al . Enhancing safety of transport by road by on-line monitoring of driver emotions [C ] // Proceedings of the 2016 11th System of Systems Engineering Conference . Piscataway:IEEE Press , 2016 : 1 - 4 .

GUO R , LI S J , HE L , et al . Pervasive and unobtrusive emotion sensing for human mental health [C ] // Proceedings of the 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops . Piscataway:IEEE Press , 2013 : 436 - 439 .

ZHAN C , LI W , OGUNBONA P , et al . A real-time facial expression recognition system for online games [J ] . International Journal of Computer Games Technology , 2008 : 542918 .

LIN S , XIE J Y , YANG M Y , et al . A review of emotion recognition using physiological signals [J ] . Sensors , 2018 , 18 ( 7 ): 2074 .

SNOEK C G M , WORRING M , SMEULDERS A W M . Early versus late fusion in semantic video analysis [C ] // Proceedings of the 13th annual ACM International Conference on Multimedia . New York:ACM Press , 2005 : 399 - 402 .

HASSAN M M , ALAM M G R , UDDIN M Z , et al . Human emotion recognition using deep belief network architecture [J ] . Information Fusion , 2019 , 51 : 10 - 18 .

KWON Y H , SHIN S B , KIM S D . Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system [J ] . Sensors , 2018 , 18 ( 5 ): 1383 .

LIAO J X , ZHONG Q H , ZHU Y S , et al . Multimodal physiological signal emotion recognition based on convolutional recurrent neural network [J ] . IOP Conference Series Materials Science and Engineering , 2020 , 782 : 032005 .

QIU L , LIU W , LYU B L . Multi-view emotion recognition using deep canonical correlation analysis [C ] // Proceedings of the 25th International Conference on Neural Information Processing . Cham:Springer , 2018 : 221 - 231 .

ZHAO Y X , CAO X Y , LIN J L , et al . Multimodal emotion recognition model using physiological signals [J ] . arXiv preprint , 2019 ,arXiv:1911.12918.

HUANG H P , HU Z C , WANG W M , et al . Multimodal emotion recognition based on ensemble convolutional neural network [J ] . IEEE Access , 2019 , 8 : 3265 - 3271 .

HOWARD A G , ZHU M L , CHEN B , et al . MobileNets:efficient convolutional neural networks for mobile vision applications [J ] . arXiv preprint , 2017 ,arXiv:1704.04861.

CHOLLET F , . Xception:deep learning with depthwise separable convolutions [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2017 : 1251 - 1258 .

LAWHERN V J , SOLON A J , WAYTOWICH N R , et al . EEGNet:a compact convolutional neural network for EEG-based brain-computer interfaces [J ] . Journal of Neural Engineering , 2018 , 15 ( 5 ): 056013 .

KOELSTRA S , MUHL C , SOLEYMANI M , et al . DEAP:a database for emotion analysis; using physiological signals [J ] . IEEE Transactions on Affective Computing , 2011 , 3 ( 1 ): 18 - 31 .

YANG Y L , WU Q F , QIU M , et al . Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network [C ] // Proceedings of the 2018 International Joint Conference on Neural Networks . Piscataway:IEEE Press , 2018 : 1 - 7 .

LIN W Q , LI C , SUN S Q . Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal [C ] // Proceedings of the International Conference on Image and Graphics . Cham:Springer , 2017 : 385 - 394 .

MA J X , TANG H , ZHENG W L , et al . Emotion recognition using multimodal residual LSTM network [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York:ACM Press , 2019 : 176 - 183 .

浏览量

1352

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于脑机协同智能的情绪识别

情感脑机接口研究综述