cs.AI提交于 2026-10-28已译

LiveSearch: 将问题作为可执行验证器代码，用于专业来源金融搜索智能体的实时基准测试

LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents

Sinuo Wang · Wentao Zhao · Jiacheng Lu · Rui Sun · Beidi Luan · Qiming Shi · et al.

中文摘要

金融领域中，价格、利率和披露信息仅对特定日期有效，获取最新事实是金融智能体的核心能力。然而，大多数金融基准测试忽略了时效性，答案在标注时固定，随市场变动而过时，且一旦泄露到训练数据中，会奖励记忆而非搜索分析。LiveSearch 是一个实时基准测试，将每个问题存储为可执行验证器代码：包含对专业金融数据服务的类型化调用及声明式提取器。问题保持冻结，黄金答案通过评估时执行程序确定，每日任务自动重新执行，无需人工或语言模型介入。构建过程同样自动化，语言模型仅将枚举自真实工具响应的程序转化为文本，从不选择答案。在通用网络搜索下，六个当代工具调用模型的答案准确率为 70.9% 至 76.7%。连续三天内，70% 的服务问题黄金值发生变化，即使采用容差匹配，直接复用前一日快照在单日内也会使多数问题失败。单问题追踪揭示了被整体准确率掩盖的正确性动态变化。

关键要点

01现有金融基准测试忽略时效性，固定答案随市场变动过时，且易被训练数据泄露利用。
02提出 LiveSearch 基准，将问题存为可执行验证器代码，评估时动态执行以获取当前黄金答案。
03构建过程自动化，语言模型仅将程序转化为文本，不直接选择答案，确保数据新鲜度。
04六个工具调用模型在通用搜索下准确率为 70.9%-76.7%，70% 问题黄金值每日变化。
05直接复用前一日快照在单日内会导致多数问题失败，单问题追踪揭示整体准确率掩盖的动态变化。

解读

尚无解读。

原始英文摘要

arXiv:2610.37166v1 Announce Type: new Abstract: In finance, the answer that matters is usually the current one: prices, rates, and disclosures hold only as of a given day, so retrieving up-to-date facts is a core capability of a financial agent. Yet most financial benchmarks overlook timeliness. They fix each answer at annotation time, so the gold goes stale as markets move and, once leaked into training data, rewards recall rather than search and analysis. We introduce LiveSearch, a live benchmark that stores each question as executable verifier code: a typed program of calls to a professional financial data service, followed by a declarative extractor. The question is frozen, while its gold answer is determined by executing the program against professional financial data at evaluation time, and a daily job re-executes it with no human or language model in the loop. A memorized answer is thus likely outdated, and a model must reach the current one through tools and analysis. Construction is equally automatic: a language model only verbalizes programs enumerated from real tool responses and never chooses an answer. Under general web search, six contemporary tool-calling models reach 70.9% to 76.7% answer accuracy. Over three consecutive days, 70% of served questions change their gold value, and even under tolerance-aware matching, a literally reused earlier snapshot fails on a majority of questions within a single day. Per-question tracking uncovers dynamic shifts in correctness masked by aggregate accuracy.