cs.AI提交于 2026-10-31已译

TreasuryQA Bench：AI 智能体能否处理扫描财务记录档案？

TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?

Ritiz Tambi · Sahar Shayegan · Joshua Meyer · Sun Kim · Ali Khan · Victor Shih · et al.

中文摘要

组织将数十年的财务记录以扫描 PDF 形式保存在共享驱动器上，AI 智能体现在可通过模型上下文协议（MCP）服务器访问这些驱动器。现有基准测试未能反映这一生产环境，因为它们简化了任务：有些将相关文档直接交给智能体，使其无需搜索；另一些则让智能体在预提取文本的文档中搜索，使其无需读取扫描图像。TreasuryQA Bench 同时要求两者：70 个问题覆盖 1,435 份美国财政部扫描出版物，存放于 Veris 的高保真 Dropbox 副本中。每个任务中，智能体通过 Dropbox 的 MCP 服务器搜索相关文件，使用 Reducto 的托管 OCR 服务器或自有工具读取所需页面，并回复一个精确数字。Python 评分器将每个答案与正确值附近的可接受范围进行比对。本文呈现十个智能体的结果：最佳为 Claude Code 中的 Claude Opus 5.5，解决 64.8% 的尝试；最差为 Codex 中的 GPT-5.6 Terra，解决 16.7%，基准远未饱和。还测试了扫描质量的影响：所有智能体在最模糊扫描件上解决的任务更少。多数失败尝试回复错误数字而非无数字，最常见原因是扫描件数字误读。附三个带答案键的示例。

关键要点

01现有基准简化任务，或直接给文档、或用预提取文本，未模拟真实扫描档案搜索场景。
02TreasuryQA Bench 含 70 题、1,435 份扫描美国财政部文件，要求智能体搜索并读取扫描页后回复精确数字。
03十个智能体中最佳 Claude Opus 5.5 解决 64.8%，最差 GPT-5.6 Terra 仅 16.7%，基准远未饱和。
04扫描质量显著影响表现：所有智能体在最模糊扫描件上解决任务更少。
05多数失败回复错误数字而非无数字，最常见原因是扫描件数字误读。

解读

尚无解读。

原始英文摘要

arXiv:2610.38924v1 Announce Type: new Abstract: Organisations keep decades of financial records as scanned PDFs on shared drives, and AI agents can now reach those drives through Model Context Protocol (MCP) servers. Existing benchmarks do not mirror this production setting because they simplify the task. In some, the relevant documents are handed to the agent, so it never has to search. In others, the agent searches through documents with pre-extracted text, so it never has to read scanned images. We introduce TreasuryQA Bench, which requires both: 70 questions over 1,435 scanned U.S. Treasury publications, held in Veris's high-fidelity Dropbox twin. In each task the agent searches for the relevant file through Dropbox's MCP server, reads the pages it needs with Reducto's hosted OCR server or its own tools, and replies with one exact figure. A Python grader checks each answer against an accepted range around the correct value. In this paper we present results from ten agents. The best, Claude Opus 5.5 in Claude Code, solves 64.8% of attempts and the worst, GPT-5.6 Terra in Codex, solves 16.7%, so the benchmark is far from saturated. We also test how scan quality affects performance: every agent solves fewer tasks on the blurriest scans. Most failed attempts reply with a wrong number rather than no number, and the most common cause is a digit misread from the scan. Three worked examples are included with their answer keys.