LLM 研究员
我要你充当大型语言模型(Large Language Model)研究领域的专家。请仔细阅读用户提供的论文、文本或概念性术语,然后回答他们提出的问题。在作答时,请确保不遗漏任何重要细节。基于你的理解,你还应说明该概念背后的原理、实现过程及目的。如有可能,你可利用网络搜索查找有关该概念或其推理过程的额外信息。在呈现信息时,凡有可用的论文引用或链接,均须一并提供。
Reliable Evaluation of AI Agents for Investment Research: A Framework for Performance and Validity Assessment
Dongnuan Tian · Yan Yu · Yangfan Liu · Wenjun Tang · Junting Kan · Xiang Wang · et al.
评估承担投资研究任务的 AI 智能体(agent)时,智能体需与外部信息源和计算工具交互,产出财务分析与研究报告。关键挑战在于:智能体最终输出的质量未必反映其执行过程的质量,而评估结果的可靠性又取决于评估流程本身的有效性。此外,投资研究任务在数值准确性、信息检索、财务分析和支撑证据等方面的要求差异很大,难以用统一标准衡量智能体表现。为此提出一个面向领域的评估框架,将结果质量与执行质量区分开,并通过任务特定标准、确定性校验器(deterministic validator)和独立的大语言模型评审来适应上述差异。同时引入评估有效性机制,检查评估输入、参考信息、支撑证据和评分输出的完整性,并由专家复核解决发现的缺陷和重大评分分歧。数值结果表明,该框架能区分智能体在不同评估维度上的表现差异,同时识别评估过程中的潜在可靠性问题。相关实现与评估材料将在 GitHub 上公开。
尚无解读。
arXiv:2611.22338v1 Announce Type: new Abstract: We consider the problem of evaluating AI agents undertaking investment-research tasks, where agents interact with external information sources and computational tools to produce financial analyses and research reports. An important challenge is that the quality of an agent's final output does not necessarily reflect the quality of its execution process, while the reliability of the resulting assessment also depends on the validity of the evaluation procedure. Furthermore, investment-research tasks differ considerably in their requirements for numerical accuracy, information retrieval, financial analysis and supporting evidence, making it difficult to assess agent performance using a uniform set of criteria. We propose a domain-oriented evaluation framework that distinguishes outcome quality from execution quality and accommodates these differences through task-specific criteria, deterministic validators and independent large language model judges. We also introduce an evaluation-validity mechanism that examines the integrity of evaluation inputs, reference information, supporting evidence and grading outputs, with expert review used to resolve identified deficiencies and significant grading disagreements. Numerical results show that the proposed framework can distinguish differences in agent performance across evaluation dimensions, while identifying potential reliability issues in the evaluation process. The associated implementation and evaluation materials will be made available on GitHub.