学习读取事件:用冻结语言模型实现精确状态跟踪
Learning to Read Events for Exact State Tracking with Frozen Language Models
Jeonghoon Lee (Attractor Dynamics Inc.)
中文摘要
长事件流需要识别每次变化并按顺序应用。在冻结语言模型上训练选择器来读取每个转换。仅通过 API 访问时,要求模型列出转换,外部整数寄存器按顺序组合它们。置信度规则可标记不确定更新或保留多个候选。研究五个项目的排列乘积任务,在 TC⁰ ≠ NC¹ 的标准猜想下,固定深度、对数精度 transformer 无法对所有输入长度求解。在两事件窗口上,27B 模型无可见推理时 100 题答对 3 题,生成中间步骤时答对 96 题。冻结的 0.8B 模型配合训练好的选择器,在 299 个窗口(每个含 43,687 个事件)中全部正确读取。同一套件在新机器上通过 299 个新窗口中的 298 个,未达到单侧 95% 置信下界 0.99 的目标。另一次允许一次失败的后续实验得到 473 个中的 472 个。在 1,000 事件的房间交换文档上,十个模型中有三个仅靠推理解决了全部五份文档。在此长度下以两种方式测试的七个模型均通过提取和寄存器解决了全部五份,未要求思维链。并发提取时,Sonnet 5.5 的中位墙钟时间从 136.1 秒降至 10.4 秒,GPT-6.1 Sol 从 313.5 秒降至 28.0 秒。固定每 token 费率下,每文档平均成本 Sonnet 5.5 降低 55%,GPT-6.1 Sol 降低 44%。带执行器的正则表达式也解决了全部 15 份固定格式文档。在 500 条自由格式问题跟踪评论上,单独适配的 4B 读取器在 68.8% 的案例中识别出记录值,而同一训练项目上拟合的频率基线为 61.0%。
关键要点
- 01冻结语言模型加训练选择器与外部整数寄存器,可精确跟踪长事件流状态。
- 0227B 模型无推理仅答对 3/100,生成中间步骤后达 96/100。
- 030.8B 模型在 299 个窗口(各 43,687 事件)全部正确,新机器通过 298/299。
- 04并发提取使 Sonnet 5.5 中位耗时从 136.1 秒降至 10.4 秒,成本降 55%。
- 05自由格式评论上 4B 读取器识别率 68.8%,高于频率基线 61.0%。
解读
尚无解读。
原始英文摘要
arXiv:2611.25211v1 Announce Type: new Abstract: Long event streams require identifying each change and applying the changes in order. We train a selector on a frozen language model to read each transition. With API-only access, we ask the model to list the transitions. An external integer register composes them in order. A confidence rule can flag uncertain updates or retain several candidates. We study products of permutations of five items, a task that fixed-depth, log-precision transformers cannot solve on all input lengths under the standard conjecture that $\mathrm{TC}^0 \ne \mathrm{NC}^1$. On two-event windows, a 27B model answered 3 of 100 correctly without visible reasoning and 96 of 100 when it generated intermediate steps. A frozen 0.8B model with a trained selector read every event in 299 of 299 windows of 43,687 events each. The same package passed 298 of 299 new windows on a fresh machine, missing the target of 0.99 for the one-sided 95% lower confidence bound. A separate follow-up sized to allow one failure yielded 472 of 473. On 1,000-event room-swap documents, three of ten models solved all five documents by reasoning alone. All seven models tested both ways at this length solved all five with extraction and the register, with no chain of thought requested. With concurrent extraction, median wall-clock time fell from 136.1 to 10.4 s for Sonnet 5.5 and from 313.5 to 28.0 s for GPT-6.1 Sol. At fixed per-token rates, mean cost per document fell by 55% for Sonnet 5.5 and 44% for GPT-6.1 Sol. A regular expression with an executor also solved all 15 fixed-format documents. On 500 free-form issue-tracker comments, a separately adapted 4B reader identified the recorded value in 68.8% of cases, compared with 61.0% for a frequency baseline fitted on the same training projects.