跳到正文
返回论文列表
cs.CL提交于 已译

盲点会迁移,知识留原地:RL 从 LLM 评判器继承了什么

Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge

Liang Yue · Eric A. Vance

中文摘要

在没有标准答案的场景中,LLM 评判器(judge)被用作奖励信号,期望其知识能传递给被训练的模型。在 814 次强化学习(RL)实验中,训练 7-8B 模型回答闭卷问题或选择弃答,我们考察究竟什么被传递。评判器对单个答案的判断不会迁移:被评判器奖励的错误答案只会被记住那一道题。用标准答案或远更强的评判器训练出的模型,在选择该回答哪些新问题时,并不比未训练模型通过采样 32 个答案并在分歧处弃答做得更好;评判器的知识大约每百题只增加一个被纠正的答案。评判器对某类问题的看法会迁移:一个拒绝了关于前国家首都的 12 个正确训练答案中 11 个的评判器,会教模型在新问题上拒绝 98% 的此类首都题,而准确率和弃答率无法揭示这一损失。仅手动改动这 12 个判定,就能把拒绝率从 5% 移到 90%,即使评判器的宽松程度与标准答案相当;在这种宽松度下,四个评判器中有两个已经损失了 38-42% 的此类首都题。这些首都知识并未被遗忘:在屏蔽“我不知道”后,模型能正确回答其中 94%。因此评判器本身无需具备知识:奖励模型给出其自身未训练采样中多数一致的答案,无需标签和评判器,即可获得与标准答案相同的奖励并保留类别能力。

关键要点

  1. 01814 次 RL 实验显示,评判器对单个答案的知识几乎不迁移,每百题仅纠正约一个答案。
  2. 02评判器对某类问题的偏见会迁移:拒绝 11/12 个首都题正确训练答案,导致新题拒绝率达 98%。
  3. 03手动改动 12 个判定可将拒绝率从 5% 调到 90%,且准确率与弃答率无法揭示该损失。
  4. 04屏蔽“我不知道”后模型能正确回答 94% 的首都题,说明知识未被遗忘。
  5. 05无需标签和评判器,用模型自身未训练采样的多数一致答案作奖励,即可达到标准答案的奖励效果并保留类别能力。

解读

尚无解读。

原始英文摘要

arXiv:2611.25191v1 Announce Type: cross Abstract: LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal. Changing only those twelve verdicts by hand moves the refusal from 5% to 90%, even for a grader that accepts as many answers as the answer key; at that leniency, two of four judges already cost 38-42% of these capitals. The capitals are not forgotten: with "I don't know" blocked, the model answers 94% of them correctly. The grader therefore needs no knowledge of its own: rewarding the model for giving the answer that most of its own untrained samples agree on, with no labels and no judge, earns the answer key's reward and keeps the category.

同方向论文 · cs.CL

查看全部 →