cs.CV提交于 2026-11-02已译

更好的评论者对于分布匹配蒸馏至关重要

A Better Critic Matters for Distribution Matching Distillation

Zongze Wu (Adobe) · Xianbang Wang (MIT) · Zhengyang Geng (Adobe) · Mingdeng Cao (Adobe) · Zhoutong Zhang (Adobe) · Richard Zhang (Adobe) · et al.

中文摘要

分布匹配蒸馏(Distribution Matching Distillation, DMD)利用教师模型的得分与在线评论者(critic)对学生分布得分的估计之间的差异,训练少步学生模型。研究评论者在容量和信息两方面需要具备什么条件,才能提供可靠的训练信号。在容量方面,在 ImageNet 上跨七种 SiT 模型尺寸的受控实验显示,固定小型评论者而增大教师模型会降低学生质量并导致训练崩溃;让评论者与教师同步扩展(称为联合扩展(joint scaling))可防止崩溃,并使学生从更大的教师中获益。在信息方面,不再强制评论者建模与步数无关的学生输出混合,而是引入步感知评论者(step-aware critic, STAC),以学生的生成步为条件估计步特定的得分。联合扩展与 STAC 结合,在类条件 ImageNet 生成、使用 SDXL 和 FLUX.2 的文本到图像生成,以及使用 Wan2.1 的文本到视频生成中,均持续改善少步蒸馏。

关键要点

01发现固定小型评论者而增大教师模型会降低学生质量并导致训练崩溃
02提出联合扩展策略,让评论者与教师同步扩展,防止崩溃并受益于更大教师
03引入步感知评论者 STAC,以生成步为条件估计步特定得分,替代步无关混合建模
04在 ImageNet、SDXL、FLUX.2 和 Wan2.1 上验证方法持续改善少步蒸馏

解读

尚无解读。

原始英文摘要

arXiv:2610.42135v1 Announce Type: cross Abstract: Distribution Matching Distillation (DMD) trains a few-step student using the difference between a teacher's score and an online critic's estimate of the student distribution's score. We investigate what the critic needs, in capacity and in information, to provide a reliable training signal. On the capacity side, controlled experiments across seven SiT model sizes on ImageNet show that increasing teacher size while keeping a small critic fixed can degrade student quality and cause training collapse. Scaling the critic alongside the teacher, a strategy we call joint scaling, prevents collapse in our experiments and allows the student to benefit from larger teachers. On the information side, rather than forcing the critic to model a step-agnostic mixture of student outputs, we introduce a step-aware critic (STAC) that conditions on the student's generation step to estimate step-specific scores. Together, joint scaling and STAC consistently improve few-step distillation across class-conditional ImageNet generation, text-to-image generation with SDXL and FLUX.2, and text-to-video generation with Wan2.1.