跳到正文
返回论文列表
cs.CV提交于 已译

MVRCap: 基于分层多维可验证奖励的全面视频描述生成

MVRCap: Towards Comprehensive Video Captioning via Hierarchical Multi-Dimensional Verifiable Rewards

Weihuang Lin · Xianing Chen · Zhibin Wang · Congzhi Zhang · Yiwei Ma · Jiayi Ji · et al.

中文摘要

当前视频描述模型受限于粗粒度监督,难以捕捉时空动态及逻辑意图。MVRCap 将视频描述重构为由结构化验证引导的强化学习任务,核心是细粒度可验证奖励机制,以强制视觉与语义对齐。该奖励系统包含三个支柱:利用新建 MVRCap-QA 数据集构建的分层概念覆盖奖励,确保多维完整性与语义深度;提升事实一致性的事实性验证奖励;以及最大化语义密度的简洁性约束。此外,论文构建了包含六个关键维度结构化标注的高质量 MVRCap-SFT 数据集,作为强化学习策略的稳健初始化基础。在 VDC、CAPability 和 VC4VG 等权威基准上的实验表明,MVRCap 取得了最先进的性能。

关键要点

  1. 01针对现有模型粗粒度监督导致无法捕捉时空动态和逻辑意图的问题,提出 MVRCap 框架。
  2. 02将视频描述重构为强化学习任务,利用细粒度可验证奖励机制实现严格的视觉语义对齐。
  3. 03构建包含分层概念覆盖、事实性验证和简洁性约束的三维奖励系统,并新建 MVRCap-QA 数据集。
  4. 04创建包含六个维度结构化标注的 MVRCap-SFT 数据集,作为强化学习策略的稳健初始化基础。
  5. 05在 VDC、CAPability 和 VC4VG 等权威基准测试中,MVRCap 实现了最先进的性能表现。

解读

尚无解读。

原始英文摘要

arXiv:2610.36838v1 Announce Type: new Abstract: Despite significant progress, current models struggle to generate comprehensive video captions, constrained primarily by coarse-grained supervision, which often fails to capture spatial-temporal dynamics and the underlying logical intent. To address this issue, we introduce MVRCap, a novel framework that reformulates video captioning as a reinforcement learning task steered by structured verification. Central to our approach is a Fine-Grained Verifiable Reward mechanism, serving as dense supervision to enforce rigorous visual-semantic alignment. Specifically, we construct this reward system anchored by three complementary pillars: a Hierarchical Concept Coverage Reward utilizing our newly constructed MVRCap-QA dataset to ensure multi-dimensional completeness and semantic depth, a Factuality Verification Reward to enhance factual consistency, and a Conciseness Constraint to maximize semantic density. Furthermore, we curate the high-quality MVRCap-SFT dataset, providing structured annotations across six key dimensions, to serve as a robust foundational initialization for the RL policy in this training pipeline. Extensive experiments demonstrate that MVRCap achieves state-of-the-art performance on authoritative benchmarks including VDC, CAPability, and VC4VG.

同方向论文 · cs.CV

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联提示词 · 1

视频github.com/f/awesome-chatgpt-prompts·外部收录·

image to video 360 product rotaion

{ "model": "veo-3.1", "task": "image_to_video_360_product_rotation", "objective": "Generate a photorealistic, silent, 360-degree rotation video from the provided front and back images of the exact same product. Preserve 100% of the o…