三思：Metis-1.6 技术报告
Think Twice: The Metis-1.6 Technical Report
Giulianno Vollmer
中文摘要
Metis 押注的是：更好的数据、更好的训练方法和新架构，能以更低成本换来更大参数量买不到的效果。Metis-1.6 是其最新一步：由一位研究者从零构建的两个语言模型，Praxis（2.94B 参数，每次前向 0.46B 激活）和 Logos（10.23B 参数，每次前向 1.18B 激活）。两者对每个 token 都运行两遍混合 Mamba-2 与注意力（attention）的专家混合（mixture-of-experts）堆栈，使每个 token 获得二次审视。每个模型共训练 1.02T token：804.76B 预训练、18B 上下文扩展至 131,072 token、200B 监督微调。Metis-1.6 从未打算挑战前沿模型，在 40 个后训练基准上大多也没有：Logos 平均 52.93，在 17 个对比模型中排第七，与 Qwen3-4B 相差不到一分。但其训练数据约为披露数据的对比模型中位数的十二分之一，且每 token 激活参数少于任何胜过它的模型，因此在两者上都处于帕累托前沿（Pareto frontier）。数学上在 30 个模型中排第二，AIME 2025 得 67.50%，HMMT February 2026 得 63.64%。它在代码、指令遵循和长上下文方面较弱。报告其余部分是我认为最有价值的内容：出了什么问题。我配置的优化器从未运行，注意力在整个预训练期间用 AdamW 训练。一个 1.77B 参数的 n-gram 记忆学会了自我关闭。五个微调选择损害了能力。在组均值归一化下，长度奖励在组通过率超过 2/3 后惩罚正确解因过长，我证明并修复了这一符号反转，只需一行 clamp。最后给出 MoRE（Mixture of Recursive Experts）背后的 10B token 消融实验，这是 Metis-1.7 的循环架构。
关键要点
- 01问题：以更少训练数据和激活参数，探索数据、训练方法与新架构能否替代更大参数量。
- 02方法：Praxis 与 Logos 采用 Mamba-2 与注意力专家混合堆栈，每 token 运行两遍。
- 03结果：Logos 在 40 个基准平均 52.93，训练数据约为对比中位数的十二分之一，处于帕累托前沿。
- 04结果：数学基准 AIME 2025 得 67.50%、HMMT February 2026 得 63.64%，在 30 个模型中排第二。
- 05局限：代码、指令遵循和长上下文较弱；优化器未运行、n-gram 记忆自我关闭等训练问题被记录。
解读
尚无解读。
原始英文摘要
arXiv:2610.37903v1 Announce Type: cross Abstract: Metis is a bet that better data, better training methods and new architectures can buy what a bigger parameter count cannot, at a fraction of the cost. Metis-1.6 is its latest step: two language models built from scratch by a single researcher, Praxis (2.94B parameters, 0.46B active per pass) and Logos (10.23B parameters, 1.18B active per pass). Both run a hybrid Mamba-2 and attention mixture-of-experts stack twice for every token, so each token gets a second look. Each was trained on 1.02T tokens in total: 804.76B of pretraining, 18B of context extension to 131,072 tokens, and 200B of supervised fine-tuning. Metis-1.6 was never meant to challenge frontier models, and across 40 post-trained benchmarks it mostly does not: Logos averages 52.93, seventh of 17 models in its comparison set and within a point of Qwen3-4B. But it gets there on roughly a twelfth of the median training data of the comparators that disclose it, and with fewer active parameters per token than any model that beats it, which puts it on the Pareto frontier for both. In mathematics it ranks second of 30 models, with 67.50% on AIME 2025 and 63.64% on HMMT February 2026. It is weak at code, instruction following and long context. The rest of the report is the part I found most valuable: what went wrong. The optimizer I configured never ran, and attention trained with AdamW for all of pretraining. A 1.77B-parameter n-gram memory learned to switch itself off. Five fine-tuning choices cost capability. And a length reward under group-mean normalization punished correct solutions for being long once a group's pass rate passed 2/3, a sign inversion I prove and fix with a one-line clamp. I close with the 10B-token ablations behind MoRE (Mixture of Recursive Experts), the looped architecture of Metis-1.7.