基于集成学习与聚类联合标注的多模态个体情绪识别

柯善军; 聂成洋; 王钰苗; 何邦胜

doi:10.11959/j.issn.2096-6652.202401

您当前的位置：

首页 >

文章列表页 >

基于集成学习与聚类联合标注的多模态个体情绪识别

学术论文 | 更新时间：2024-06-05

- 基于集成学习与聚类联合标注的多模态个体情绪识别
- Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering
- 智能科学与技术学报 2024年6卷第1期页码：76-87
- 作者机构：
  
  重庆理工大学车辆工程学院，重庆 400050
- 作者简介：
  
  [ "柯善军（1975- ），男，重庆理工大学车辆工程学院副教授、硕士生导师，主要研究方向为机电产品工业设计、用户体验与人机交互、汽车主动安全。" ]
  [ "聂成洋（2000- ），男，重庆理工大学车辆工程学院硕士生，主要研究方向为驾驶员状态监测。" ]
  [ "王钰苗（2000- ），女，重庆理工大学车辆工程学院硕士生，主要研究方向为用户体验与人机交互。" ]
  [ "何邦胜（1998- ），男，重庆理工大学车辆工程学院硕士生，主要研究方向为智能座舱测试技术。" ]
- 基金信息：
  
  重庆市教委科学技术研究项目(2020CJZ053)
- DOI：10.11959/j.issn.2096-6652.202401
  中图分类号： TP911.7;TP183
- 收稿：2023-08-31，
  
  修回：2024-01-30，
  
  纸质出版：2024-03-15
- 稿件说明：
移动端阅览
柯善军, 聂成洋, 王钰苗, 等. 基于集成学习与聚类联合标注的多模态个体情绪识别[J]. 智能科学与技术学报, 2024,6(1):76-87.

KE Shanjun, NIE Chengyang, WANG Yumiao, et al. Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(1): 76-87.
柯善军, 聂成洋, 王钰苗, 等. 基于集成学习与聚类联合标注的多模态个体情绪识别[J]. 智能科学与技术学报, 2024,6(1):76-87. DOI： 10.11959/j.issn.2096-6652.202401.

KE Shanjun, NIE Chengyang, WANG Yumiao, et al. Multimodal individual emotion recognition with joint labeling based on integrated learning and clustering[J]. Chinese Journal of Intelligent Science and Technology, 2024, 6(1): 76-87. DOI： 10.11959/j.issn.2096-6652.202401.

摘要

针对通用情绪识别模型面对不同个体时的低识别精度问题，提出一种基于集成学习与聚类联合标注的多模态个体情绪识别方法。该方法首先基于公共数据集训练通用情绪识别模型，然后分析公共数据集数据与个体无标签数据的分布差异，建立跨域模型来预测和标注个体数据的伪标签。同时，对个体数据进行加权聚类并标注聚类标签，利用聚类标签与伪标签进行联合标注，筛选高置信度样本进一步训练通用模型，得到个性化情绪识别模型。实验采集3名被试的3种情绪数据并使用该方法标注，最后优化得到的个性化模型对3种情绪的平均识别精度达到80%以上，相比原通用模型，至少提升了35%。

Abstract

To address the low recognition accuracy of generic emotion recognition models when faced with different individuals

a multimodal individual emotion recognition technique based on joint labelling with integrated learning and clustering was proposed. The method first trained a generic emotion recognition model based on a public dataset

then anallysed the distributional differences between the data in the public dataset and the unlabelled data of individuals

and established a cross-domain model for predicting and labelling pseudo-labels of individual data. At the same time

the individual data were weighted clustered and labelled with cluster labels

and the cluster labels were used to jointly label with pseudo-labels

and high confidence samples were screened to further train the generic model to obtain a personalized emotion recognition model. Using this method to annotate these data with the experimentally collected data of 3 emotions from 3 subjects

the final optimized personalized model achieved an average recognition accuracy of more than 80% for the 3 emotions

which was at least a 35% improvement compared to the original generic model.

关键词

Keywords

references

殷梦馨, 倪娜, 尉怀怀, 等. 基于多模态情绪识别的研究进展[J]. 生物医学工程研究, 2023, 42(3): 285-291.

YIN M X, NI N, YU H H, et al. Research progress based on multimodal emotion recognition[J]. Journal of Biomedical Engineering Research, 2023, 42(3): 285-291.

CHEN M, ZHOU P, FORTINO G. Emotion communication system[J]. IEEE Access, 2016, 5: 326-337.

ZEPF S, HERNANDEZ J, SCHMITT A, et al. Driver emotion recognition for intelligent vehicles: a survey[J]. ACM Computing Surveys, 2020, 53(3): 64.

XIA Y X, LIU Y H. EEG-based emotion recognition with consideration of individual difference[J]. Sensors, 2023, 23(18): 7749.

ZHAO J M, LI R C, JIN Q, et al. Memobert: pre-training model with prompt-based learning for multimodal emotion recognition[C]//Proceedings of the ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Piscataway: IEEE Press, 2022: 4703-4707.

LEE D H. Pseudo-Label: the simple and efficient semi-supervised learning method for deep neural networks[C]//Proceedings of International Conference on Machine Learning.[S.l.:s.n.], 2013.

SOHN K, BERTHELOT D, LI C L, et al. FixMatch: simplifying semi-supervised learning with consistency and confidence[C]//Proceedings of the Proceedings of the 34th International Conference on Neural Information Processing Systems. New York: ACM, 2020: 596-608.

LI H Y, WANG N N, YANG X, et al. Towards semi-supervised deep facial expression recognition with an adaptive confidence margin[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2022: 4156-4165.

ZHU Z, SATO Y. Speech emotion recognition using semi-supervised learning with efficient labeling strategies[C]//Proceedings of the 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). Piscataway: IEEE Press, 2021: 358-365.

ZHANG S, CHEN M, CHEN J C, et al. Combining cross-modal knowledge transfer and semi-supervised learning for speech emotion recognition[J]. Knowledge-Based Systems, 2021, 229: 107340.

CHEN M, HAO Y X. Label-less learning for emotion cognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(7): 2430-2440.

ZHANG Z X, HAN J, DENG J, et al. Leveraging unlabeled data for emotion recognition with enhanced collaborative semi-supervised learning[J]. IEEE Access, 2018, 6: 22196-22209.

李学聪, 战荫伟, 杨卓, 等. 基于标签分布学习的眼部情绪识别[J]. 计算机科学与应用, 2022(4): 1213-1225.

LI X C, ZHAN Y W, YANG Z, et al. Eye emotion recognition based on label distribution learning[J]. Computer Science and Application, 2022(4): 1213-1225.

SOLEYMANI M, LICHTENAUER J, PUN T, et al. A multimodal database for affect recognition and implicit tagging[J]. IEEE Transactions on Affective Computing, 2012, 3(1): 42-55.

CHEN G H, ZENG X P. Multi-modal emotion recognition by fusing correlation features of speech-visual[J]. IEEE Signal Processing Letters, 2021, 28: 533-537.

ZHANG Y, CHENG C, ZHANG Y D. Multimodal emotion recognition using a hierarchical fusion convolutional neural network[J]. IEEE Access, 2021, 9: 7943-7951.

SUBRAMANIAN G, CHOLENDIRAN N, PRATHYUSHA K, et al. Multimodal emotion recognition using different fusion techniques[C]//Proceedings of the 2021 Seventh International conference on Bio Signals, Images, and Instrumentation (ICBSII). Piscataway: IEEE Press, 2021: 1-6.

凌文芬, 陈思含, 彭勇, 等. 基于3D分层卷积融合的多模态生理信号情绪识别[J]. 智能科学与技术学报, 2021, 3(1): 76-84.

LING W F, CHEN S H, PENG Y, et al. Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 76-84.

CHEN Y S, JIA Z Y, HIROTA K, et al. A multimodal emotion perception model based on context-aware decision-level fusion[C]//Proceedings of the 2022 41st Chinese Control Conference (CCC). Piscataway: IEEE Press, 2022: 7332-7337.

ZHAO Y X, CAO X Y, LIN J L, et al. Multimodal affective states recognition based on multiscale CNNs and biologically inspired decision fusion model[J]. IEEE Transactions on Affective Computing, 2023, 14(2): 1391-1403.

PAN B, HIROTA K, JIA Z Y, et al. A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methods[J]. Neurocomputing, 2023, 561: 126866.

GUYON I, WESTON J, BARNHILL S, et al. Gene selection for cancer classification using support vector machines[J]. Machine Learning, 2002, 46(1): 389-422.

LAWHERN V J, SOLON A J, WAYTOWICH N R, et al. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces[J]. Journal of Neural Engineering, 2018, 15(5): 056013.

MURUGESAN B, RAVICHANDRAN V, RAM K, et al. ECGNet: deep network for arrhythmia classification[C]//Proceedings of the 2018 IEEE International Symposium on Medical Measurements and Applications (MeMeA). Piscataway: IEEE Press, 2018: 1-6.

SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB]. arXiv preprint, 2014, arXiv: 1409.1556.

GOODFELLOW I J, ERHAN D, CARRIER P L, et al. Challenges in representation learning: a report on three machine learning contests[J]. Neural Networks, 2015, 64: 59-63.

高寒冰, 张驰, 靳明艳, 等. 基于多源域自适应的跨被试情感脑电识别[J]. 智能科学与技术学报, 2021, 3(1): 59-64.

GAO H B, ZHANG C, JIN M Y, et al. Cross-subject emotional EEG recognition based on multi-source domain adaptation[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 59-64.

吕宝粮, 张亚倩, 郑伟龙. 情感脑机接口研究综述[J]. 智能科学与技术学报, 2021, 3(1): 36-48.

LU B L, ZHANG Y Q, ZHENG W L. A survey of affective brain-computer interface[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 36-48.

FARAHANI A, VOGHOEI S, RASHEED K, et al. A brief review of domain adaptation[C]//Proceedings of Advances in Data Science and Information Engineering. Cham: Springer, 2021: 877-894.

GLOROT X, BORDES A, BENGIO Y. Domain adaptation for large-scale sentiment classification: a deep learning approach[C]//Proceedings of the Proceedings of the 28th International Conference on International Conference on Machine Learning. New York: ACM, 2011: 513-520.

柴鑫. 基于领域适配的跨个体脑电情绪识别方法研究[D]. 哈尔滨: 哈尔滨工业大学, 2018.

CHAI X. Research on cross-individual EEG emotion recognition method based on domain adaptation[D]. Harbin: Harbin Institute of Technology, 2018.

SALAMA E S, EL-KHORIBI R A, SHOMAN M E, et al. A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition[J]. Egyptian Informatics Journal, 2021, 22(2): 167-176.

胡毅, 瞿博阳, 梁静, 等. 进化集成学习算法综述[J]. 智能科学与技术学报, 2021, 3(1): 18-33.

HU Y, QU B Y, LIANG J, et al. A survey on evolutionary ensemble learning algorithm[J]. Chinese Journal of Intelligent Science and Technology, 2021, 3(1): 18-33.

CARDONE B, DI MARTINO F, SENATORE S. Improving the emotion‐based classification by exploiting the fuzzy entropy in FCM clustering[J]. International Journal of Intelligent Systems, 2021, 36(11): 6944-6967.

徐鹏飞, 黄宇霞, 罗跃嘉. 中国情绪影像材料库的初步编制和评定[J]. 中国心理卫生杂志, 2010, 24(7): 551-554, 561.

XU P F, HUANG Y X, LUO Y J. Establishment and assessment of native Chinese affective video system[J]. Chinese Mental Health Journal, 2010, 24(7): 551-554, 561.

浏览量

378

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于选择性深度嵌入聚类的复杂未知雷达信号分选方法

进化集成学习算法综述

Cerberus：基于深度学习的跨站社交网络机器人检测系统