跳到正文
返回论文列表
cs.LG提交于 已译

跨问题可靠强化学习

Cross-Question Reliable Reinforcement Learning

Hector G. Rodriguez · Marcus Rohrbach

中文摘要

在能力边界附近运行的 AI 系统,可通过弃答或将不确定的问题转交人类或更强模型,在最大化效用的同时最小化错误风险。近期,大语言模型(LLM)经过后训练以估计自身答案的置信度。现有方法基于每个问答对的正确性信号单独强化置信度分数,奖励机制鼓励将正确答案的置信度设为 1、错误答案设为 0。然而,置信度估计的价值在于仅选择最自信的答案,以在满足用户风险容忍度的同时最大化已解决问题比例。将正确答案置信度设为 1、错误答案设为 0 在理论上最优,但实践中不可达,且对非零风险容忍度的选择性预测有害:置信度被推向 0 和 1 后不再携带答案间相对比较的信息。为此提出 CARE(跨问题弃答奖励估计),在一组问答对上计算奖励:在若干风险容忍度下,集合内的置信度诱导出弃答阈值,每个置信度因位于阈值正确一侧而获得奖励。由于阈值本身来自集合,每个言语化置信度是相对于其他问题的置信度被强化,而非对照自身的二元正确性标签。在数学任务上,CARE 将 MATH-500、GSM8K 和 Big-Math-Digits 在 5% 风险下的覆盖率(C@5)从 0.8% 提升至 42.4%,优于现有的逐点正确性匹配奖励。在 HotpotQA 的多跳维基百科问答上训练后,CARE 在分布外通用知识、推理和数学基准(包括 CommonsenseQA、GPQA、SimpleQA 和 TriviaQA)上将 C@5 从 12.3% 提升至 23.6%。

关键要点

  1. 01现有逐点置信度奖励将正确/错误答案的置信度推向 1/0,导致非零风险容忍下选择性预测失效
  2. 02CARE 在问答集合上按多个风险容忍度诱导弃答阈值,奖励置信度位于阈值正确一侧
  3. 03数学任务上 C@5 从 0.8% 提升至 42.4%,显著优于逐点正确性匹配奖励
  4. 04HotpotQA 训练后,分布外通用知识、推理和数学基准的 C@5 从 12.3% 提升至 23.6%

解读

尚无解读。

原始英文摘要

arXiv:2611.25292v1 Announce Type: new Abstract: AI systems operating near the limits of their capabilities can maximize utility while minimizing the risk of error by abstaining or deferring questions they are not confident about to a human or a stronger model. Recently, large language models (LLMs) have been post-trained to estimate confidence in their own answers. To achieve this, methods reinforce confidence scores based on the correctness signal to each question-answer pair individually, with rewards that encourage assigning confidence 1 to correct answers and 0 to incorrect answers. However, estimating confidence scores is useful in that it enables selecting only the most confident answers, to maximize the share of solved questions while accommodating the user's risk tolerance. Assigning confidence 1 to correct answers and 0 to incorrect ones is theoretically optimal, but unattainable in practice and detrimental for selective prediction with a non-zero risk tolerance: confidences pushed to 0 and 1 carry no information about how answers compare to one another. We therefore introduce CARE (Cross-question Abstention Reward Estimation), which computes rewards over a set of question-answer pairs: At each of several risk tolerances, the confidences across the set induce an abstention threshold, and each confidence is rewarded for lying on the correct side of it. Since the threshold itself comes from the set, every verbalized confidence is reinforced relative to other questions' confidence, not against its own binary correctness label. On math, CARE increases the coverage at 5% risk (C@5) across MATH-500, GSM8K, and Big-Math-Digits from 0.8% to 42.4%, compared to existing pointwise correctness-matching rewards. When trained on multi-hop Wikipedia QA from HotpotQA, CARE improves C@5 in out-of-distribution general-knowledge, reasoning, and math benchmarks (including CommonsenseQA, GPQA, SimpleQA, and TriviaQA) from 12.3% to 23.6%.

同方向论文 · cs.LG

查看全部 →