跳到正文
返回论文列表
cs.AI提交于 已译

模型自适应工具必要性揭示 LLM 工具使用的知行差距

Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use

Yize Cheng · Chenrui Fan · Mahdi JafariRaviz · Keivan Rezaei · Soheil Feizi

中文摘要

大语言模型(LLM)日益作为自主智能体运行,需要判断应直接回答还是调用外部工具。既有研究通常将工具必要性视为与模型无关的属性,交由人类或 LLM 评判者标注,且多聚焦答案显而易见的情形。然而,由于不同模型的能力边界存在差异,实际场景中的工具必要性更复杂:强模型可独立解决的问题,对较弱模型而言仍可能需要工具。研究依据各模型的实测表现,定义了模型自适应的工具必要性,并在四个模型的算术与事实问答数据集上,将其与实际工具调用行为进行比较;两类不匹配比例分别达到 26.5–54.0% 和 30.8–41.8%。为诊断失败原因,工具使用被分解为反映模型是否认为需要工具的内部认知阶段,以及决定模型是否实际调用工具的执行阶段。探测 LLM 隐状态发现,两类信号通常均可线性解码,但在驱动下一 token 动作的末层、末 token 区间,其探针方向几乎正交。轨迹追踪进一步表明,多数不匹配集中在从认知到行动的转换,而非认知本身。结果显示,LLM 工具使用存在知行差距:提高可靠性不仅需要更好识别何时需要工具,也需将这种认知更好地转化为行动。

关键要点

  1. 01既有工具必要性定义忽略模型能力边界差异,难以准确刻画实际工具需求。
  2. 02依据模型实测表现定义模型自适应工具必要性,并与真实工具调用行为进行比较。
  3. 03算术与事实问答中的不匹配比例分别达到 26.5–54.0% 和 30.8–41.8%。
  4. 04工具使用可拆分为内部认知与执行阶段,两类信号在末层末 token 区间几乎正交。
  5. 05多数不匹配源于认知到行动的转换,暴露 LLM 工具使用的知行差距。

解读

尚无解读。

原始英文摘要

arXiv:2605.14038v3 Announce Type: replace Abstract: Large language models (LLMs) increasingly act as autonomous agents that must decide when to answer directly vs. when to invoke external tools. Prior work studying adaptive tool use has largely treated tool necessity as a model-agnostic property, annotated by human or LLM judge, and mostly cover cases where the answer is obvious (e.g., fetching the weather vs. paraphrasing text). However, tool necessity in the wild is more nuanced due to the divergence of capability boundaries across models: a problem solvable by a strong model on its own may still require tools for a weaker one. In this work, we introduce a model-adaptive definition of tool-necessity, grounded in each model's empirical performance. Following this definition, we compare the necessity against observed tool-call behavior across four models on arithmetic and factual QA dataset, and find substantial mismatches of 26.5-54.0% and 30.8-41.8%, respectively. To diagnose the failure, we decompose tool use into two stages: an internal cognition stage that reflects whether a model believes a tool is necessary, and an execution stage that determines whether the model actually makes a tool-call action. By probing the LLM hidden states, we find that both signals are often linearly decodable, yet their probe directions become nearly orthogonal in the late-layer, last-token regime that drives the next-token action. By tracing the trajectory of samples in the two-stage process, we further discover that the majority of mismatch is concentrated in the cognition-to-action transition, not in cognition itself. These results reveal a knowing-doing gap in LLM tool-use: improving tool-use reliability requires not only better recognition of when tools are needed, but also better translation of that recognition into action.

同方向论文 · cs.AI

查看全部 →

反向发现 · 这篇论文相关的工具与提示词

去交叉发现 →

关联提示词 · 6

研究github.com/CHENJIAMIAN/awesome-chatgpt-prompts-zh-CN·外部收录·

LLM 研究员

我要你充当大型语言模型(Large Language Model)研究领域的专家。请仔细阅读用户提供的论文、文本或概念性术语,然后回答他们提出的问题。在作答时,请确保不遗漏任何重要细节。基于你的理解,你还应说明该概念背后的原理、实现过程及目的。如有可能,你可利用网络搜索查找有关该概念或其推理过程的额外信息。在呈现信息时,凡有可用的论文引用或链接,均须一并提供。

智能体github.com/f/awesome-chatgpt-prompts·外部收录·

Weekend Planner

- Read the weekend weather, calendar and budget. - Find local events, hikes, exhibitions and restaurants. - Propose 3 plans: adventure, cozy, social, with times and costs. - Prepare reservation steps; book nothing without approval. - Rememb…