LLMs 不知道自己知道什么：面向金融回测的时间自知识基准测试
LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting
Lingfei Qian · Yan Wang · Xueqing Peng · Yupeng Cao · Yuechen Jiang · Zhuohan Xie · et al.
中文摘要
只有当模型并不已知被要求预测的结果时，对大型语言模型（LLM）的金融回测才有意义。从业者通常依赖供应商声明知识截止日期、询问模型自身知道什么，或指示其以过去某日期作答来防范。为验证这些防护措施是否可信，我们推出 FinDendro，一个包含 1999 至 2026 年间 2,226 条带日期美国事实的基准，覆盖月度宏观经济序列、月末股价和美联储决议。每条事实生成四个提示：一个数值问题；Recall 与 Predict 问题，将每个候选值与声称知道或猜测配对且不声明日期；以及带历史声明日期的 Predict 问题。对持续性和主导结果的对照使随机准确率接近 0.27。结果显示，对多数前沿模型，自我报告低估了知识。GPT-5 在受控事实上答对 76%，但声称知道 38%，且在其声称不知道的事实中答对 61%。知识声称可超出实测知识边界，最多达七个季度；声明日期消除了大部分先前知识声称，但准确率不变。因此，模型训练窗口内的回测无法通过自我报告、指令或声明截止日期来认证无泄漏。这些发现推动对时间知识边界进行行为测量，FinDendro 提供了基于已证明事实知识选择回测周期的基准与流程。
关键要点
- 01现有防护（供应商截止日期、模型自报、指定日期作答）无法可靠防止金融回测中的知识泄漏。
- 02FinDendro 含 1999–2026 年 2,226 条带日期美国事实，随机准确率控制在约 0.27。
- 03GPT-5 在受控事实上答对 76%，却仅声称知道 38%，且答对其声称不知道事实的 61%。
- 04知识声称可超出实测边界最多七个季度；声明日期消除声称但不改变准确率。
- 05训练窗口内回测无法凭自报、指令或截止日期认证无泄漏，需行为测量知识边界。
解读
尚无解读。
原始英文摘要
arXiv:2611.06808v1 Announce Type: new Abstract: Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.