校准决策奖励:用知道自身确定性的评判器训练语言模型
Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is
Rudransh Agnihotri · Manasvi Kapoor
中文摘要
强化学习(RL)只能教会语言模型其奖励函数能识别的内容。答案检查器精确但需要参考答案;学习到的奖励模型和LLM评判器随处可用,但其分数并非概率。研究RLCDR(基于校准决策奖励的强化学习)。将LLM转换为决策模型,对响应回答类型化问题(“最终答案是否正确?”,“哪个步骤是第一个错误?”),通过一次前向传播读取概率分布,对选项顺序去偏,并用约一百个标签校准。在MATH数据集上用GRPO训练Qwen3-1.7B,这种无参考奖励将MATH500准确率从72.2%提升至76.8%(+4.65点,p < 1e-4),与用答案键训练(76.9%)相当,而随机奖励仅提升+0.65。增益随难度增加,在最难级别从44.8%升至51.7%。在策略自身变化的样本上,奖励保持校准:其期望校准误差是相同27B骨干上softmax评判器的一半(0.074 vs. 0.141),且无奖励黑客迹象。无需步骤训练,同一模型在寻找第一个错误步骤上优于专用的7B过程奖励模型(ProcessBench F1 0.772 vs. 0.722,20/20分割)。还描绘了该方法不足的边界:仅在基模型大多失败的问题上训练,无参考奖励仍能改进策略(+1.9 MATH500点,p = 0.03)但落后于答案键,因为其概率在该分布上变得过度自信。预先陈述六个假设,并用旨在证伪的对照逐一测试。
关键要点
- 01问题:学习奖励模型和LLM评判器的分数非概率,无法可靠指导RL训练。
- 02方法:将LLM转为决策模型,输出概率分布并校准,作为无参考奖励用于GRPO训练。
- 03结果:MATH500准确率从72.2%升至76.8%,与答案键训练相当,且校准误差更低。
- 04结果:在ProcessBench上F1达0.772,优于专用7B过程奖励模型。
- 05局限:在基模型失败的问题上训练时,奖励过度自信,改进落后于答案键。
解读
尚无解读。
原始英文摘要
arXiv:2611.25135v1 Announce Type: cross Abstract: Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.