跳到正文
返回论文列表
cs.AI提交于 已译

强化学习驱动的扩散模型:语言、视觉及更多领域

Reinforcement Learning-driven Diffusion Models: Language, Vision, and Beyond

Zhiyuan Ma · Jiabao Wei · Tianyu Wang · Hanqing Wang · Jiadong Pan · Haowen Xu · et al.

中文摘要

扩散模型(DMs)在“预训练-微调”范式下展现出高保真生成能力,但在复杂推理、自主探索及对齐人类目标方面存在局限。受 RLHF 启发,强化学习(RL)成为通过奖励驱动轨迹优化迭代改进 DMs 的关键后训练机制。该综述首次全面梳理 RL 驱动的 DMs,分析其如何重塑扩散过程以实现以经验为中心的目标导向生成。文章构建了涵盖语言、多模态、机器人及科学模型的技术全景,重点探讨扩散轨迹优化与奖励工程。通过多任务基准实验,系统评估代表性方法以展示不同 RL 策略的优势。研究揭示了阻碍实际部署的三大瓶颈:奖励设计脆弱性、高维生成空间中的探索-利用权衡,以及稀疏奖励下的训练不稳定性。最后,提供了将 RL 集成到 DMs 的原则性路线图,并展望了构建鲁棒、可解释的 RL-扩散系统的未来方向。

关键要点

  1. 01指出传统扩散模型在复杂推理、自主探索及对齐人类目标方面存在关键局限。
  2. 02提出技术全景,阐述 RL 在语言、多模态、机器人及科学模型中的多重角色。
  3. 03通过多任务基准实验,系统评估不同 RL 后训练策略的优势与表现。
  4. 04揭示奖励设计脆弱性、探索-利用权衡及稀疏奖励下训练不稳定三大瓶颈。
  5. 05提供 RL 集成路线图,并建立公开仓库持续追踪该领域最新进展。

解读

尚无解读。

原始英文摘要

arXiv:2610.21539v1 Announce Type: new Abstract: Diffusion models (DMs) have steadily demonstrated outstanding performance in high-fidelity, controllable generation across diverse scenarios, following the "pretraining-finetuning" paradigm established in LLMs. Despite these advances, conventional DMs still face critical limitations in sophisticated reasoning, autonomous exploration of generation pathways, and adaptability to complex, human-aligned objectives. Inspired by RLHF's success in LLMs, RL has recently emerged as a pivotal post-training mechanism to iteratively refine DMs through reward-driven trajectory optimization. This paper presents the first comprehensive review of RL-driven DMs and an empirical analysis of how RL reshapes the diffusion process towards experience-centric, goal-directed generation. We propose a novel technical panorama elucidating RL's multifaceted roles in diffusion-based language, multimodal, robotic, and scientific models, with a granular focus on diffusion trajectory optimization and reward engineering. Through extensive experiments on multiple task-specific benchmarks, we systematically evaluate representative methods to intuitively highlight the advantages of different RL post-training strategies. Furthermore, we reveal and dissect three crucial bottlenecks that currently hinder the real-world deployment of RL-driven DMs: reward design fragility, exploration-exploitation trade-off in high-dimensional generation spaces, and training instability under sparse rewards. Based on these perspectives, we provide a principled roadmap for the integration of RL into DMs and outline promising future research directions for robust, interpretable RL-diffusion systems. We also construct a public repository to continuously track the latest developments in this rapidly evolving field: https://github.com/MAIR-Lab-HUST/Awesome-RL-for-Diffusion-Models.

同方向论文 · cs.AI

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联工具 · 1

Microsoft Research67

微软 CARE-X:结合辅助监督、对齐强化学习和工具增强测量的临床可用胸部 X 光视觉语言模型

微软研究院发布 CARE-X,一个统一胸部 X 光 VLM,结合生成和判别能力,使用 DAPO 强化学习进行临床对齐优化,并配备辅助分类头和接地头提供校准概率和空间定位。

推荐理由:强化学习与辅助监督的结合在医学影像 VLM 中展现了临床价值,工具增强测量在需要定量诊断的条件上提升了平均 F1 达 43.6 个百分点。