cs.AI提交于 2026-10-31已译

仅从参考样本学习推理：面向现代 LLM 后训练的强化对抗学习

Learning to Reason from Reference Samples Alone: Reinforced Adversarial Learning for Modern LLM Post-Training

Albert Bou · Javier Valdes-Aleman · Jon M. Laurent · Samuel G. Rodriques · Andrew D. White · Michaela Hinks · et al.

中文摘要

在许多需要 LLM 后训练的场景中，高质量答案可被识别，却难以用显式奖励打分。可验证奖励强化学习（RLVR）仅适用于存在明确正确性信号的任务。此前的对抗式推理 LLM 训练方法，要么用协同训练的评判器补充可验证奖励，要么需要参考推理轨迹。本文探究：协同训练的 LLM 评判器能否仅凭参考答案，构成推理 LLM 在策略后训练的唯一奖励。该方法称为强化对抗学习（RAL）。先在三个可验证任务（整数乘法、基于代码的算术表达式合成、SMILES 到分子式翻译）上验证 RAL，验证器仅用于评估。在评判器足够强且训练充分时，RAL 可匹配直接 RLVR。随后将 RAL 应用于两个场景的科学摘要生成：其一，提示未指定领域，RAL 引导生成器产出参考摘要所属科学领域的摘要；其二，以资助标题为条件，为美国 National Institutes of Health 资助申请书生成摘要。人类科学家与 LLM 评判器集成认为，RAL 生成结果比前沿 LLM 基线更难与真实摘要区分。这表明，在科学等高价值领域，可借助日益强大的评判器从示例答案推断难以指定的奖励函数，从而后训练推理模型。

关键要点

01现有 RLVR 依赖显式正确性信号，难以覆盖高质量答案可识别但难打分的任务。
02RAL 用协同训练的 LLM 评判器，仅以参考答案作为在策略后训练的唯一奖励。
03在整数乘法、算术表达式合成、SMILES 翻译三个可验证任务上，RAL 可匹配直接 RLVR。
04在 NIH 资助摘要生成中，RAL 输出比前沿 LLM 基线更难被人类与 LLM 评判器区分。
05局限在于需要足够强的评判器与充分训练，才能逼近可验证奖励的效果。

解读

尚无解读。

原始英文摘要

arXiv:2610.38851v1 Announce Type: new Abstract: In many settings where LLM post-training is desirable, high-quality answers are recognizable but difficult to score with explicit rewards. Reinforcement learning with verifiable rewards (RLVR) is limited to tasks where an explicit correctness signal is available. Prior adversarial approaches to training reasoning LLMs either complement verifiable rewards with co-trained critics or require reference reasoning traces. We ask whether a co-trained LLM judge can turn reference answers alone into the sole reward for on-policy post-training of reasoning LLMs. We call this method Reinforced Adversarial Learning (RAL). We first validate RAL on three verifiable tasks (integer multiplication, code-based arithmetic-expression synthesis, and SMILES-to-formula translation), using the verifiers only for evaluation. With sufficiently capable judges and enough training, RAL matches direct RLVR. We then apply RAL to scientific abstract generation in two settings. In the first setting, RAL guides the generator to produce abstracts in the scientific domain of the reference abstracts, even though the prompt leaves that domain unspecified. In the second setting, we generate abstracts for U.S. National Institutes of Health grant proposals conditioned on the grant title. Human scientists and an ensemble of LLM judges find our generations harder to distinguish from real abstracts than those of frontier-LLM baselines. These results suggest a path toward post-training reasoning models in scientific and other high-value domains, where increasingly capable judges infer hard-to-specify reward functions from example answers.