Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion

Wenfen LING; Sihan CHEN; Yong PENG; Wanzeng KONG

doi:10.11959/j.issn.2096-6652.202108

您当前的位置：

首页 >

文章列表页 >

Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion

Special topic:emotional brain computer interface | 更新时间：2024-06-05

- Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion
- Chinese Journal of Intelligent Science and Technology Vol. 3, Issue 1, Pages: 76-84(2021)
- 作者机构：
  
  1. 杭州电子科技大学计算机学院，浙江杭州 310018
  2. 浙江省脑机协同智能重点实验室，浙江杭州 310018
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-6652.202108
  CLC： TP18，TN911.7，R318
- Published Online：2021-03，
  
  Published：15 March 2021
- 稿件说明：
移动端阅览
Wenfen LING, Sihan CHEN, Yong PENG, et al. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 76-84.
DOI：

Wenfen LING, Sihan CHEN, Yong PENG, et al. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion[J]. Chinese journal of intelligent science and technology, 2021, 3(1): 76-84. DOI： 10.11959/j.issn.2096-6652.202108.

摘要

近年来，脑电等生理信号由于能客观体现真实情绪已逐渐成为情绪识别研究的热门对象。然而，单模态的脑电信号存在情绪信息特征不完备问题，多模态生理信号存在情绪信息交互不充分问题。针对这些问题，提出基于3D分层卷积的多模态特征融合模型，旨在充分挖掘多模态交互关系，更准确地刻画情感信息。首先分别通过深度可分离卷积网络提取脑电、眼电和肌电3种模态的生理信号的多模态初级情绪特征信息，再对得到的多模态初级情绪特征信息进行3D卷积融合操作，实现两两模态间的局部交互以及所有模态间的全局交互，获取包含不同生理信号情绪特征的多模态融合特征。实验结果表明，提出的模型在DEAP数据集的效价、唤醒度的二分类和四分类任务中达到了98%的平均准确率。

Abstract

In recent years

physiological signals such as electroencephalograhpy (EEG) have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However

the single-modal EEG signal has the problem of incomplete emotional information representation

and the multi-modal physiological signal has the problem of insufficient emotional information interaction.Therefore

a 3D hierarchical convolutional fusion model was proposed

which aimed to fully explore multi-modal interaction relationships and more accurately describe emotional information.The method first extracted the primary emotional representation information of EEG

electro-oculogram (EOG) and electromyography (EMG) by depthwise separable convolution network

and then performed 3D convolution fusion operation on the obtained multi-modal primary emotional representation information to realize the pairwise mode local interactions between states and global interactions among all modalities

so as to obtain multi-modal fusion representations containing emotional characteristics of different physiological signals.The results show that the accuracy in the valence and arousal of the two-class and four-class tasks on DEAP dataset are both 98% by the proposed model.

关键词

Keywords

references

DORNAIKA F , RADUCANU B . Efficient facial expression recognition for human robot interaction [C ] // Proceedings of the 9th International Work-Conference on Artificial Neural Networks . Berlin:Springer , 2007 : 700 - 708 .

BARTNECK C , LYONS M J . HCI and the face:towards an art of the soluble [C ] // Proceedings of the 21th International Conference on Human-Computer Interaction . Berlin:Springer , 2007 : 20 - 29 .

DE NADAI S , D'INCÀ M , PARODI F , et al . Enhancing safety of transport by road by on-line monitoring of driver emotions [C ] // Proceedings of the 2016 11th System of Systems Engineering Conference . Piscataway:IEEE Press , 2016 : 1 - 4 .

GUO R , LI S J , HE L , et al . Pervasive and unobtrusive emotion sensing for human mental health [C ] // Proceedings of the 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops . Piscataway:IEEE Press , 2013 : 436 - 439 .

ZHAN C , LI W , OGUNBONA P , et al . A real-time facial expression recognition system for online games [J ] . International Journal of Computer Games Technology , 2008 : 542918 .

LIN S , XIE J Y , YANG M Y , et al . A review of emotion recognition using physiological signals [J ] . Sensors , 2018 , 18 ( 7 ): 2074 .

SNOEK C G M , WORRING M , SMEULDERS A W M . Early versus late fusion in semantic video analysis [C ] // Proceedings of the 13th annual ACM International Conference on Multimedia . New York:ACM Press , 2005 : 399 - 402 .

HASSAN M M , ALAM M G R , UDDIN M Z , et al . Human emotion recognition using deep belief network architecture [J ] . Information Fusion , 2019 , 51 : 10 - 18 .

KWON Y H , SHIN S B , KIM S D . Electroencephalography based fusion two-dimensional (2D)-convolution neural networks (CNN) model for emotion recognition system [J ] . Sensors , 2018 , 18 ( 5 ): 1383 .

LIAO J X , ZHONG Q H , ZHU Y S , et al . Multimodal physiological signal emotion recognition based on convolutional recurrent neural network [J ] . IOP Conference Series Materials Science and Engineering , 2020 , 782 : 032005 .

QIU L , LIU W , LYU B L . Multi-view emotion recognition using deep canonical correlation analysis [C ] // Proceedings of the 25th International Conference on Neural Information Processing . Cham:Springer , 2018 : 221 - 231 .

ZHAO Y X , CAO X Y , LIN J L , et al . Multimodal emotion recognition model using physiological signals [J ] . arXiv preprint , 2019 ,arXiv:1911.12918.

HUANG H P , HU Z C , WANG W M , et al . Multimodal emotion recognition based on ensemble convolutional neural network [J ] . IEEE Access , 2019 , 8 : 3265 - 3271 .

HOWARD A G , ZHU M L , CHEN B , et al . MobileNets:efficient convolutional neural networks for mobile vision applications [J ] . arXiv preprint , 2017 ,arXiv:1704.04861.

CHOLLET F , . Xception:deep learning with depthwise separable convolutions [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2017 : 1251 - 1258 .

LAWHERN V J , SOLON A J , WAYTOWICH N R , et al . EEGNet:a compact convolutional neural network for EEG-based brain-computer interfaces [J ] . Journal of Neural Engineering , 2018 , 15 ( 5 ): 056013 .

KOELSTRA S , MUHL C , SOLEYMANI M , et al . DEAP:a database for emotion analysis; using physiological signals [J ] . IEEE Transactions on Affective Computing , 2011 , 3 ( 1 ): 18 - 31 .

YANG Y L , WU Q F , QIU M , et al . Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network [C ] // Proceedings of the 2018 International Joint Conference on Neural Networks . Piscataway:IEEE Press , 2018 : 1 - 7 .

LIN W Q , LI C , SUN S Q . Deep convolutional neural network for emotion recognition using EEG and peripheral physiological signal [C ] // Proceedings of the International Conference on Image and Graphics . Cham:Springer , 2017 : 385 - 394 .

MA J X , TANG H , ZHENG W L , et al . Emotion recognition using multimodal residual LSTM network [C ] // Proceedings of the 27th ACM International Conference on Multimedia . New York:ACM Press , 2019 : 176 - 183 .

Views

1352

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Emotion recognition based on brain and machine collaborative intelligence

A survey of affective brain-computer interface

Related Author

Dongjun LIU

Yuhan WANG

Wenfen LING

Yong PENG

Wanzeng KONG

Bao-Liang LU

Yaqian ZHANG

Wei-Long ZHENG

Related Institution

College of Computer Science and Technology, Hangzhou Dianzi University

Center for Brain-like Computing and Machine Intelligence, Department of Computer Science and Engineering, Shanghai Jiao Tong University

Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University

Brain Science and Technology Research Center, Shanghai Jiao Tong University

Qing Yuan Research Institute, Shanghai Jiao Tong University

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：35685 Visits Today：160

⁰