从表达到反应：用于对话者情绪识别的角色感知视觉迁移与刺激引导推理
From Expression to Reaction: Role-aware Visual Transfer and Stimulus-guided Reasoning for Interlocutor Emotion Recognition
Wei Wang · Zhaowu Li · Jianjie Luo · Fu Lee Wang · Lap-Kei Lee · Zhenguo Yang
中文摘要
本文提出角色感知刺激引导（RASG）框架，用于对话者情绪识别。该框架从仅 listener 视频和仅 speaker 音频预测 listener 情绪。RASG 由角色感知视觉迁移（RVT）和刺激引导边界推理（SBR）模块组成，分别解决缺乏标注 listener 数据导致的监督不匹配问题，以及视觉相似的 listener 反应（其解释取决于 speaker 上下文）存在的歧义问题。具体而言，RVT 选择面部表情支持其情绪标签的 speaker 样本，过滤 listener 轨迹，并使用可靠伪标签训练以 listener 为中心的视觉专家。SBR 仅在视觉模型不确定时使用二分类语言推理器，将 speaker 音频和文本作为上下文而非直接情绪证据，以区分相似的 listener 反应。在 MER-Cross 数据集上的实验表明，RASG 达到 76.25% 准确率，比基线提高 17% 以上。团队在 ACM MM 2026 MER 大挑战赛道 1（MER-Cross）中排名第二。
关键要点
- 01问题：缺乏标注的 listener 情绪数据导致监督不匹配，且视觉相似的 listener 反应因缺乏 speaker 上下文而存在歧义
- 02方法：RVT 模块选择面部表情与情绪标签一致的 speaker 样本，为 listener 视频生成可靠伪标签训练视觉专家
- 03方法：SBR 模块仅在视觉模型不确定时调用二分类语言推理器，将 speaker 音频和文本作为上下文辅助区分相似反应
- 04结果：在 MER-Cross 数据集上达到 76.25% 准确率，比基线方法提升超过 17%
- 05局限：依赖 speaker 音频文本作为上下文信息，对纯视觉场景的泛化能力有待验证
解读
尚无解读。
原始英文摘要
arXiv:2610.03016v1 Announce Type: cross Abstract: In this paper, we propose a Role-aware Stimulus-guided (RASG) framework for interlocutor emotion recognition, which predicts listener emotions from listener-only videos and speaker-only audios. RASG consists of Role-aware Visual Transfer (RVT) and Stimulus-guided Boundary Reasoning (SBR) modules, which address supervision mismatch due to the lack of labeled listener data and ambiguity among visually similar listener reactions whose interpretation depends on speaker context, respectively. More specifically, RVT selects speaker samples whose facial expressions support their emotion labels. It then filters listener tracks and uses reliable pseudo-labels to train a listener-centric visual expert. SBR uses a two-class language reasoner only when the visual model is uncertain. It treats speaker audio and text as context rather than direct emotion evidence to distinguish similar listener reactions. Experiments conducted on MER-Cross dataset shows that RASG achieves 76.25\% on MER-Cross and improves the performance of the baseline over 17\%. Our team ranks second in Track 1 (MER-Cross) of the MER Grand Challenge at ACM MM 2026.