Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering

KE Shanjun; NIE Chengyang; WANG Yumiao; HE Bangsheng

doi:10.11959/j.issn.2096-6652.202401

您当前的位置：

首页 >

文章列表页 >

Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering

Papers and Reports | 更新时间：2024-06-05

- Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering
- Chinese Journal of Intelligent Science and Technology Vol. 6, Issue 1, Pages: 76-87(2024)
- 作者机构：
  
  重庆理工大学车辆工程学院，重庆 400050
- 作者简介：
- 基金信息：
  
  The Scientific and Technological Research Program of Chongqing Municipal Education Commission(2020CJZ053)
- DOI：10.11959/j.issn.2096-6652.202401
  CLC： TP911.7;TP183
- Received：31 August 2023，
  
  Revised：2024-01-30，
  
  Published：15 March 2024
- 稿件说明：
移动端阅览
KE Shanjun, NIE Chengyang, WANG Yumiao, et al. Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(1): 76-87.
DOI：

KE Shanjun, NIE Chengyang, WANG Yumiao, et al. Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(1): 76-87. DOI： 10.11959/j.issn.2096-6652.202401.

摘要

针对通用情绪识别模型面对不同个体时的低识别精度问题，提出一种基于集成学习与聚类联合标注的多模态个体情绪识别方法。该方法首先基于公共数据集训练通用情绪识别模型，然后分析公共数据集数据与个体无标签数据的分布差异，建立跨域模型来预测和标注个体数据的伪标签。同时，对个体数据进行加权聚类并标注聚类标签，利用聚类标签与伪标签进行联合标注，筛选高置信度样本进一步训练通用模型，得到个性化情绪识别模型。实验采集3名被试的3种情绪数据并使用该方法标注，最后优化得到的个性化模型对3种情绪的平均识别精度达到80%以上，相比原通用模型，至少提升了35%。

Abstract

To address the low recognition accuracy of generic emotion recognition models when faced with different individuals

a multimodal individual emotion recognition technique based on joint labelling with integrated learning and clustering was proposed. The method first trained a generic emotion recognition model based on a public dataset

then anallysed the distributional differences between the data in the public dataset and the unlabelled data of individuals

and established a cross-domain model for predicting and labelling pseudo-labels of individual data. At the same time

the individual data were weighted clustered and labelled with cluster labels

and the cluster labels were used to jointly label with pseudo-labels

and high confidence samples were screened to further train the generic model to obtain a personalized emotion recognition model. Using this method to annotate these data with the experimentally collected data of 3 emotions from 3 subjects

the final optimized personalized model achieved an average recognition accuracy of more than 80% for the 3 emotions

which was at least a 35% improvement compared to the original generic model.

关键词

Keywords

references

殷梦馨, 倪娜, 尉怀怀, 等. 基于多模态情绪识别的研究进展[J]. 生物医学工程研究, 2023, 42(3): 285-291.

YIN M X, NI N, YU H H, et al. Research progress based on multimodal emotion recognition[J]. Journal of Biomedical Engineering Research, 2023, 42(3): 285-291.

CHEN M, ZHOU P, FORTINO G. Emotion communication system[J]. IEEE Access, 2016, 5: 326-337.

ZEPF S, HERNANDEZ J, SCHMITT A, et al. Driver emotion recognition for intelligent vehicles: a survey[J]. ACM Computing Surveys, 2020, 53(3): 64.

XIA Y X, LIU Y H. EEG-based emotion recognition with consideration of individual difference[J]. Sensors, 2023, 23(18): 7749.

ZHAO J M, LI R C, JIN Q, et al. Memobert: pre-training model with prompt-based learning for multimodal emotion recognition[C]//Proceedings of the ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Piscataway: IEEE Press, 2022: 4703-4707.

LEE D H. Pseudo-Label: the simple and efficient semi-supervised learning method for deep neural networks[C]//Proceedings of International Conference on Machine Learning.[S.l.:s.n.], 2013.

SOHN K, BERTHELOT D, LI C L, et al. FixMatch: simplifying semi-supervised learning with consistency and confidence[C]//Proceedings of the Proceedings of the 34th International Conference on Neural Information Processing Systems. New York: ACM, 2020: 596-608.

LI H Y, WANG N N, YANG X, et al. Towards semi-supervised deep facial expression recognition with an adaptive confidence margin[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2022: 4156-4165.

ZHU Z, SATO Y. Speech emotion recognition using semi-supervised learning with efficient labeling strategies[C]//Proceedings of the 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). Piscataway: IEEE Press, 2021: 358-365.

ZHANG S, CHEN M, CHEN J C, et al. Combining cross-modal knowledge transfer and semi-supervised learning for speech emotion recognition[J]. Knowledge-Based Systems, 2021, 229: 107340.

CHEN M, HAO Y X. Label-less learning for emotion cognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(7): 2430-2440.

ZHANG Z X, HAN J, DENG J, et al. Leveraging unlabeled data for emotion recognition with enhanced collaborative semi-supervised learning[J]. IEEE Access, 2018, 6: 22196-22209.

李学聪, 战荫伟, 杨卓, 等. 基于标签分布学习的眼部情绪识别[J]. 计算机科学与应用, 2022(4): 1213-1225.

LI X C, ZHAN Y W, YANG Z, et al. Eye emotion recognition based on label distribution learning[J]. Computer Science and Application, 2022(4): 1213-1225.

SOLEYMANI M, LICHTENAUER J, PUN T, et al. A multimodal database for affect recognition and implicit tagging[J]. IEEE Transactions on Affective Computing, 2012, 3(1): 42-55.

CHEN G H, ZENG X P. Multi-modal emotion recognition by fusing correlation features of speech-visual[J]. IEEE Signal Processing Letters, 2021, 28: 533-537.

ZHANG Y, CHENG C, ZHANG Y D. Multimodal emotion recognition using a hierarchical fusion convolutional neural network[J]. IEEE Access, 2021, 9: 7943-7951.

SUBRAMANIAN G, CHOLENDIRAN N, PRATHYUSHA K, et al. Multimodal emotion recognition using different fusion techniques[C]//Proceedings of the 2021 Seventh International conference on Bio Signals, Images, and Instrumentation (ICBSII). Piscataway: IEEE Press, 2021: 1-6.

凌文芬, 陈思含, 彭勇, 等. 基于3D分层卷积融合的多模态生理信号情绪识别[J]. 智能科学与技术学报, 2021, 3(1): 76-84.

LING W F, CHEN S H, PENG Y, et al. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 76-84.

CHEN Y S, JIA Z Y, HIROTA K, et al. A multimodal emotion perception model based on context-aware decision-level fusion[C]//Proceedings of the 2022 41st Chinese Control Conference (CCC). Piscataway: IEEE Press, 2022: 7332-7337.

ZHAO Y X, CAO X Y, LIN J L, et al. Multimodal affective states recognition based on multiscale CNNs and biologically inspired decision fusion model[J]. IEEE Transactions on Affective Computing, 2023, 14(2): 1391-1403.

PAN B, HIROTA K, JIA Z Y, et al. A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methods[J]. Neurocomputing, 2023, 561: 126866.

GUYON I, WESTON J, BARNHILL S, et al. Gene selection for cancer classification using support vector machines[J]. Machine Learning, 2002, 46(1): 389-422.

LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces[J]. Journal of Neural Engineering, 2018, 15(5): 056013.

MURUGESAN B, RAVICHANDRAN V, RAM K, et al. ECGNet: deep network for arrhythmia classification[C]//Proceedings of the 2018 IEEE International Symposium on Medical Measurements and Applications (MeMeA). Piscataway: IEEE Press, 2018: 1-6.

SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB]. arXiv preprint, 2014, arXiv: 1409.1556.

GOODFELLOW I J, ERHAN D, CARRIER P L, et al. Challenges in representation learning: a report on three machine learning contests[J]. Neural Networks, 2015, 64: 59-63.

高寒冰, 张驰, 靳明艳, 等. 基于多源域自适应的跨被试情感脑电识别[J]. 智能科学与技术学报, 2021, 3(1): 59-64.

GAO H B, ZHANG C, JIN M Y, et al. Cross-subject emotional EEG recognition based on multi-source domain adaptation[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 59-64.

吕宝粮, 张亚倩, 郑伟龙. 情感脑机接口研究综述[J]. 智能科学与技术学报, 2021, 3(1): 36-48.

LU B L, ZHANG Y Q, ZHENG W L. A survey of affective brain-computer interface[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 36-48.

FARAHANI A, VOGHOEI S, RASHEED K, et al. A brief review of domain adaptation[C]//Proceedings of Advances in Data Science and Information Engineering. Cham: Springer, 2021: 877-894.

GLOROT X, BORDES A, BENGIO Y. Domain adaptation for large-scale sentiment classification: a deep learning approach[C]//Proceedings of the Proceedings of the 28th International Conference on International Conference on Machine Learning. New York: ACM, 2011: 513-520.

柴鑫. 基于领域适配的跨个体脑电情绪识别方法研究[D]. 哈尔滨: 哈尔滨工业大学, 2018.

CHAI X. Research on cross-individual EEG emotion recognition method based on domain adaptation[D]. Harbin: Harbin Institute of Technology, 2018.

SALAMA E S, EL-KHORIBI R A, SHOMAN M E, et al. A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition[J]. Egyptian Informatics Journal, 2021, 22(2): 167-176.

胡毅, 瞿博阳, 梁静, 等. 进化集成学习算法综述[J]. 智能科学与技术学报, 2021, 3(1): 18-33.

HU Y, QU B Y, LIANG J, et al. A survey on evolutionary ensemble learning algorithm[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 18-33.

CARDONE B, DI MARTINO F, SENATORE S. Improving the emotion‐based classification by exploiting the fuzzy entropy in FCM clustering[J]. International Journal of Intelligent Systems, 2021, 36(11): 6944-6967.

徐鹏飞, 黄宇霞, 罗跃嘉. 中国情绪影像材料库的初步编制和评定[J]. 中国心理卫生杂志, 2010, 24(7): 551-554, 561.

XU P F, HUANG Y X, LUO Y J. Establishment and assessment of native Chinese affective video system[J]. Chinese Mental Health Journal, 2010, 24(7): 551-554, 561.

Views

378

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A complex unknown radar signal deinterleaving method based on selective deep embedding clustering

A survey on evolutionary ensemble learning algorithm

Cerberus: a cross-site social bot detection system based on deep learning

Related Author

DING Chencong

Yi HU

Boyang QU

Jing LIANG

Jie WANG

Yanli WANG

Related Institution

Unit of PLA

School of Electrical Engineering, Zhengzhou University

School of Electronic and Information, Zhongyuan University of Technology

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：162003 Visits Today：166

⁰