展示你的工作:小型开放模型能否用机器可核验的证据读取 SEC 文件?
Show Your Work: Can Small Open Models Read SEC Filings with Evidence a Machine Can Check?
Krishna Nandula
中文摘要
语言模型给出的流畅答案并不等于经过验证的答案。本文考察运行在单张消费级 GPU 上的小型开放模型能否从 SEC 文件中提取数字,并附上机器可核验的证据:页码以及包含该数字的逐字引用。我们提出 Filing-Evidence 基准,包含 1,000 个问题,覆盖 45 家美国公司最近的 90 份 10-K 和 10-Q 文件,答案键取自申报方的 inline XBRL 标签。我们在单张 RTX 4090 上评估了来自 10 家开发商的 10 个 instruct 模型(7-21B 参数),设置三种模式:普通、检查并重试、模型自身的推理模式。在普通设置下,带可核验证据的准确率从 17.1% 到 82.7%,最佳结果为 84.5%(Qwen3.5-9B)。自动检查器会拒绝许多错误答案,但每 1,000 个答案中仍有 16 到 167 个对可回答问题给出错误数值却通过所有检查。在普通设置下,其中 57% 是比例或单位错误,30% 取错了正确行的另一个期间或列。推理模式对提供该模式的 5 个模型中的 4 个显著减少了此类错误。检查失败后重试可将准确率最多提升 3.9 个百分点,但从未降低准确率。问题、模型输出、代码和协议均已公开。
关键要点
- 01提出 Filing-Evidence 基准,含 1,000 个问题,覆盖 45 家美国公司的 90 份 SEC 文件,答案键取自 inline XBRL 标签。
- 02在单张 RTX 4090 上评估 10 个 7-21B 参数的小型开放模型,普通设置下带证据准确率为 17.1% 至 82.7%,最佳为 Qwen3.5-9B 的 84.5%。
- 03自动检查器仍会放过每 1,000 个答案中 16 至 167 个错误数值,其中 57% 为比例或单位错误,30% 取错期间或列。
- 04推理模式对 5 个模型中的 4 个显著减少上述错误;检查失败后重试最多提升准确率 3.9 个百分点,且从不降低。
解读
尚无解读。
原始英文摘要
arXiv:2611.24990v1 Announce Type: new Abstract: A fluent answer from a language model is not a verified one. We ask whether small open models that run on a single consumer GPU can extract numbers from SEC filings with evidence a machine can check: the page and a verbatim quote containing the number. We introduce Filing-Evidence, a benchmark of 1,000 questions over 90 recent 10-K and 10-Q filings of 45 US companies, with answer keys taken from the filers' inline XBRL tags. We evaluate ten instruct models from ten developers (7-21B parameters) on one RTX 4090 in three setups: plain, check-and-retry, and the model's own reasoning mode. Accuracy with checkable evidence ranges from 17.1% to 82.7% in the plain setup, and the best result is 84.5% (Qwen3.5-9B). An automatic checker rejects many wrong answers, but between 16 and 167 answers per 1,000 give a wrong value for an answerable question and still pass every check. In the plain setup, 57% of these are scale or unit errors and 30% take another period or column of the right line. Reasoning significantly reduces such errors for 4 of 5 models that offer it. Retrying after a failed check raises accuracy by up to 3.9 points but never reduces them. Questions, model outputs, code and protocol are public.