SpecCTMC：扩散语言模型连续时间马尔可夫链的精确投机模拟
SpecCTMC: Exact Speculative Simulation of Diffusion Language Model Continuous-Time Markov Chains
Siva Rajesh Kasa · Yasong Dai · Sumit Negi · Hongdong Li
中文摘要
掩码扩散语言模型支持并行文本生成，但独立采样多个 token 会破坏其依赖关系并改变输出分布。精确连续时间采样可避免这一近似，但通常每个 token 需要一次顺序模型评估。SpecCTMC 是一种无需训练的投机采样器，无需单独的草稿模型即可缓解该顺序瓶颈。它利用先前计算的模型预测提出连续 token 更新，并行评估其中间状态，并应用精确的接受与修正流程。证明表明，SpecCTMC 保持底层连续时间马尔可夫链的事件时间、生成顺序与 token 值的联合分布，因此不引入时间离散化误差，也不引入并行解码带来的额外因子化误差。在 LLaDA-8B-Instruct 与 Dream-v0-Instruct-7B 上跨五个基准的实验显示，相比直接连续时间采样，顺序模型评估次数减少 1.73-4.38 倍，同时保持参考采样分布。代码见 https://github.com/kasakh/specctmc。
关键要点
- 01问题：掩码扩散语言模型并行采样多个 token 会破坏依赖并改变输出分布。
- 02方法：无需草稿模型的投机采样，复用已有预测并行评估中间状态并精确接受修正。
- 03结果：证明保持事件时间、生成顺序与 token 值的联合分布，无离散化或因子化误差。
- 04结果：在 LLaDA-8B-Instruct 与 Dream-v0-Instruct-7B 上顺序评估减少 1.73-4.38 倍。
解读
尚无解读。
原始英文摘要
arXiv:2611.08362v1 Announce Type: new Abstract: Masked diffusion language models offer parallel text generation, but independently sampling multiple tokens can disrupt their dependencies and change the output distribution. Exact continuous-time sampling avoids this approximation, yet typically requires one sequential model evaluation per token. We introduce SpecCTMC, a training-free speculative sampler that reduces this sequential bottleneck without a separate draft model. SpecCTMC proposes successive token updates using previously computed model predictions, evaluates their intermediate states in parallel, and applies an exact acceptance-and-correction procedure. We prove that SpecCTMC preserves the joint distribution of event times, generation order, and token values of the underlying continuous-time Markov chain. It therefore introduces neither time-discretization error nor additional factorization error from parallel decoding. Experiments on LLaDA-8B-Instruct and Dream-v0-Instruct-7B across five benchmarks demonstrate 1.73-4.38x fewer sequential model evaluations than direct continuous-time sampling, while preserving the reference sampling distribution. Our code is available at: https://github.com/kasakh/specctmc