A cross-modal emergency recognition method integrating attention collaboration and contrastive learning

HUANG Shaonian; PENG Yongtao; WEN Peiran; LIU Yao

doi:10.11959/j.issn.2096-6652.202521

您当前的位置：

首页 >

文章列表页 >

A cross-modal emergency recognition method integrating attention collaboration and contrastive learning

Papers and Reports | 更新时间：2025-11-18

- A cross-modal emergency recognition method integrating attention collaboration and contrastive learning
- Chinese Journal of Intelligent Science and Technology Vol. 7, Issue 3, Pages: 338-349(2025)
- 作者机构：
  
  1.湖南工商大学计算机学院，湖南长沙 410205
  2.统计学习与智能计算湖南省重点实验室，湖南长沙 410205
  3.湖南工商大学数字媒体工程与人文学院，湖南长沙 410205
- 作者简介：
- 基金信息：
  
  The National Social Science Foundation of China(21BTJ026)
- DOI：10.11959/j.issn.2096-6652.202521
  CLC： TP391
- Received：13 February 2025，
  
  Revised：2025-04-28，
  
  Published：15 September 2025
- 稿件说明：
移动端阅览
黄少年,彭永涛,文沛然等.融合注意力协同和对比学习的跨模态突发事件识别方法[J].智能科学与技术学报,2025,07(03):338-349.

HUANG Shaonian,PENG Yongtao,WEN Peiran,et al.A cross-modal emergency recognition method integrating attention collaboration and contrastive learning[J].Chinese Journal of Intelligent Science and Technology,2025,07(03):338-349.
黄少年,彭永涛,文沛然等.融合注意力协同和对比学习的跨模态突发事件识别方法[J].智能科学与技术学报,2025,07(03):338-349. DOI： 10.11959/j.issn.2096-6652.202521.

HUANG Shaonian,PENG Yongtao,WEN Peiran,et al.A cross-modal emergency recognition method integrating attention collaboration and contrastive learning[J].Chinese Journal of Intelligent Science and Technology,2025,07(03):338-349. DOI： 10.11959/j.issn.2096-6652.202521.

摘要

针对跨模态突发事件识别中图像复杂、文本有限以及模态间误导信息干扰等问题，提出了一种融合注意力协同和对比学习的跨模态突发事件识别方法。首先，通过构建融合多支路自注意力的图像特征编码器和融合外部知识的文本特征编码器，分别提取图像的深层语义特征和文本的上下文关联信息，以克服单一模态信息表达的局限性。其次，为有效捕捉图像-文本特征间的复杂关联，设计跨模态注意力协同网络以增强模态间特征的交互一致性，减少误导信息的传递。最后，引入对比学习机制，采用联合监督策略优化图像-文本特征的贡献，实现模态间信息的动态平衡。在公开数据集CrisisMMD和自建数据集上开展的实验证明了所提方法的优越性。

Abstract

To address the challenges of image complexity

limited textual information

and inter-modal misleading data in cross-modal emergency recognition

a cross-modal emergency recognition method that integrates attention collaboration and contrastive learning was proposed. First

a multi-branch self-attention mechanism was integrated into an image feature encoder

while a text feature encoder was augmented with external knowledge. These encoders were designed to extract rich semantic features from images and contextual information from text

respectively

to overcome the limitations of single-modal representations. Next

a cross-modal attention collaboration network was introduced to capture intricate relationships between image-text features

enhancing inter-modal consistency and mitigating the influence of misleading information. Finally

a contrastive learning mechanism

which was optimized through a joint supervision strategy

was employed to dynamically balance the contributions of image and text features. Extensive experiments conducted on the public CrisisMMD dataset and the self-constructed dataset demonstrate the superior performance of the proposed method.

关键词

Keywords

references

管泽礼, 杜军平, 薛哲, 等. 基于强化联邦GNN的个性化公共安全突发事件检测[J]. 软件学报, 2024, 35(4): 1774-1789.

GUAN Z L, DU J P, XUE Z, et al. Personalized public safety event detection based on reinforcement federated GNN[J]. Journal of Software, 2024, 35(4): 1774-1789.

KUMAR A, SINGH J P, RANA N P, et al. Multi-channel convolutional neural network for the identification of eyewitness tweets of disaster[J]. Information Systems Frontiers, 2023, 25(4): 1589-1604.

周红磊, 张海涛, 栾宇, 等. 基于文本-图像增强的突发事件识别及分类方法研究[J]. 情报理论与实践, 2024, 47(4):181-188.

ZHOU H L, ZHANG H T, LUAN Y, et al. Research on emergencies identification and classification method based on text-image enhancement[J]. Information Studies (Theory & Application), 2024, 47(4):181-188.

WANG Z H, GUO Y, WANG J H. Empower Chinese event detection with improved atrous convolution neural networks[J]. Neural Computing and Applications, 2021, 33(11): 5805-5820.

LI J K, WANG Y H, LI W X. MGMP: multimodal graph message propagation network for event detection[C]//Proceeding of the International Conference on Multimedia Modeling. Cham: Springer International Publishing, 2022: 141-153.

陈宏, 钱胜胜, 李章明, 等. 基于多模态掩码Transformer网络的社会事件分类[J]. 北京航空航天大学学报, 2024, 50(2): 579-587.

CHEN H, QIAN S S, LI Z M, et al. Multi-modal mask Transformer network for social event classification[J]. Journal of Beijing University of Aeronautics and Astronautics, 2024, 50(2): 579-587.

KOSHY R, ELANGO S. Multimodal tweet classification in disaster response systems using transformer-based bidirectional attention model[J]. Neural Computing and Applications, 2023, 35(2): 1607-1627.

MUJAHID M, KANWAL K, RUSTAM F, et al. Arabic ChatGPT tweets classification using RoBERTa and BERT ensemble model[J]. ACM Transactions on Asian and Low-Resource Language Information Processing, 2023, 22(8): 1-23.

YANDRAPATI P B, ESWARI R. Classifying informative tweets using feature enhanced pre-trained language model[J]. Social Network Analysis and Mining, 2024, 14(1): 48.

ZHOU H, YIN H P, ZHENG H Y, et al. A survey on multi-modal social event detection[J]. Knowledge-Based Systems, 2020, 195: 105695.

BROUSMICHE M, ROUAT J, DUPONT S. Multimodal attentive fusion network for audio-visual event recognition[J]. Information Fusion, 2022, 85: 52-59.

KUMAR A, SINGH J P, DWIVEDI Y K, et al. A deep multi-modal neural network for informative Twitter content classification during emergencies[J]. Annals of Operations Research, 2022, 319(1): 791-822.

SIRBU I, SOSEA T, CARAGEA C, et al. Multimodal semi-supervised learning for disaster tweet classification[C]//Proceedings of the 29th International Conference on Computational Linguistics. Piscataway: IEEE Press, 2022: 2711-2723.

MADICHETTY S, SRIDEVI M. A novel method for identifying the damage assessment tweets during disaster[J]. Future Generation Computer Systems, 2021, 116: 440-454.

MADICHETTY S, SRIDEVI M. A stacked convolutional neural network for detecting the resource tweets during a disaster[J]. Multimedia Tools and Applications, 2021, 80(3): 3927-3949.

UPADHYAY A, MEENA Y K, CHAUHAN G S. SatCoBiLSTM: self-attention based hybrid deep learning framework for crisis event detection in social media[J]. Expert Systems with Applications, 2024, 249: 123604.

ALAM F, OFLI F, IMRAN M. Processing social media images by combining human and machine computing during crises[J ] . International Journal of Human - Computer Interaction, 2018, 34(4): 311-327.

VALDEZ D B, GODMALIN R A G. A deep learning approach of recognizing natural disasters on images using convolutional neural network and transfer learning[C]//Proceedings of the International Conference on Artificial Intelligence and Its Applications. New York: ACM, 2021: 1-7.

KYRKOU C, THEOCHARIDES T. Deep-learning-based aerial image classification for emergency response applications using unmanned aerial vehicles[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE Press, 2019: 517-525.

NALLURU G, PANDEY R, PUROHIT H. Relevancy classification of multimodal social media streams for emergency services[C]//Proceedings of the 2019 IEEE International Conference on Smart Computing (SMARTCOMP). Piscataway: IEEE Press, 2019: 121-125.

KIELA D, BHOOSHAN S, FIROOZ H, et al. Supervised multimodal bitransformers for classifying images and text[EB]. arXiv preprint, 2019, arXiv: 1909.02950.

VIELZEUF V, LECHERVY A, PATEUX S, et al. CentralNet: a multilayer approach for multimodal fusion[C]//Proceedings of the Computer Vision-ECCV 2018 Workshops. Cham: Springer International Publishing, 2019: 575-589.

AREVALO J, SOLORIO T, MONTES-Y-GÓMEZ M, et al. Gated multimodal networks[J]. Neural Computing and Applications, 2020, 32(14): 10209-10228.

LI L H, YATSKAR M, YIN D, et al. VisualBERT: a simple and performant baseline for vision and language[EB]. arXiv preprint, 2019, arXiv: 1908.03557.

HUANG Z C, ZENG Z Y, LIU B, et al. Pixel-BERT: aligning image pixels with text by deep multi-modal transformers[EB]. arXiv preprint, 2020, arXiv: 2004.00849.

KIM W, SON B, KIM I. ViLT: vision-and-language transformer without convolution or region supervision[C]//Proceedings of the International conference on machine learning. New York: PMLR, 2021: 5583-5594.

FUKUI A, PARK D H, YANG D, et al. Multimodal compact bilinear pooling for visual question answering and visual grounding[EB]. arXiv preprint, 2016, arXiv: 1606.01847.

WANG J, YANG S, ZHAO H. Crisis event summary generative model based on hierarchical multimodal fusion[J]. Pattern Recognition, 2023, 144: 109890.

LYU J D, WANG X G, SHAO C L. AMAE: adversarial multimodal auto-encoder for crisis-related tweet analysis[J]. Computing, 2023, 105(1): 13-28.

ABAVISANI M, WU L W, HU S L, et al. Multimodal categorization of crisis events in social media[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2020: 14679-14689.

WU X H, MAO J, XIE H, et al. Identifying humanitarian information for emergency response by modeling the correlation and independence between text and images[J]. Information Processing & Management, 2022, 59(4): 102977.

GUPTA S, SAINI N, KUNDU S M, et al. CrisisKAN: knowledge-infused and explainable multimodal attention network for crisis event classification[C]//Advances in Information Retrieval. Cham: Springer Nature Switzerland, 2024: 18-33.

MAO Y D, JIANG Q P, CONG R M, et al. Cross-modality fusion and progressive integration network for saliency prediction on stereoscopic 3D images[J]. IEEE Transactions on Multimedia, 2021, 24: 2435-2448.

CLARK K, LUONG M T, LE Q V, et al. ELECTRA: pre-training text encoders as discriminators rather than generators[EB]. arXiv preprint, 2020, arXiv: 2003.10555.

ALAM F, OFLI F, IMRAN M. CrisisMMD: multimodal twitter datasets from natural disasters[J]. arXiv preprint, 2018, arXiv: 1805.00713.

DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint, 2019, arXiv: 1810.04805.

REZK M, ELMADANY N, HAMAD R K, et al. Categorizing crises from social media feeds via multimodal channel attention[J ] . IEEE Access, 2023, 11: 72037-72049.

LIANG T, LIN G S, WAN M Y, et al. Expanding large pre-trained unimodal models with multimodal information injection for image-text multimodal classification[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE Press, 2022: 15471-15480.

SUSLADKAR O, DESHMUKH G, MAKWANA D, et al. GAFNet: a global fourier self attention based novel network for multi-modal downstream tasks[C]//Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE Press, 2023: 5231-5240.

Views

123

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Cerberus: a cross-site social bot detection system based on deep learning

Related Author

No data

Related Institution

No data

Postal code：100079
Tel：（010）53879206 Email：cjist@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：161982 Visits Today：145

⁰