记账不是判断:EJBench,一个由共轭传热设计活动构建的工程判断基准
Bookkeeping Is Not Judgement: EJBench, an Engineering Judgement Benchmark Built from a Conjugate Heat Transfer Design Campaign
James Sadler · Rotimi Alabi · Kotub Uddin
中文摘要
EJBench 是一个工程判断基准,考察模型能否根据少量先前结果预测设计在新工况下的行为并据此决策。与测试智能体能否操作仿真或 CAD 软件的工程基准不同,EJBench 不依赖特定软件,只问工程师在不运行仿真时会得出什么结论。任务来自电池冷板的共轭传热设计活动,分三类:将无标签测试结果匹配到其运行工况、在签核设计前选择揭示哪些已存运行、按峰值电芯温度对新任务排序。所有答案取自收敛的仿真,而非 LLM 评判。十个前沿模型中最好的 GPT-6 Astra 得分为 0.71,且模型在答错任务上给出的置信度几乎与答对任务相同。还用 LoRA 和强化学习对 Qwen3.5-9B 进行后训练,奖励为仿真结果确认的答案标准占比;在留出测试集上得分从 0.071 升至 0.106。发布训练、验证和测试划分、评估工具及排行榜。
关键要点
- 01EJBench 考察模型能否从少量历史结果预测新工况下的设计行为并决策,不依赖仿真或 CAD 软件操作。
- 02任务来自电池冷板共轭传热设计活动,分匹配工况、选择揭示运行、按峰值温度排序三类。
- 03十个前沿模型中最佳 GPT-6 Astra 得分为 0.71,且模型在答错任务上置信度与答对时几乎相同。
- 04用 LoRA 和强化学习后训练 Qwen3.5-9B,在留出测试集上得分从 0.071 升至 0.106。
解读
尚无解读。
原始英文摘要
arXiv:2611.21707v1 Announce Type: cross Abstract: We introduce EJBench, a benchmark of engineering judgement: whether a model can predict how a design behaves at new conditions from a few earlier results, and decide on it. Unlike engineering benchmarks that test whether an agent can operate simulation or CAD software, EJBench is software-agnostic and asks what an engineer would conclude without running a simulation. Its tasks come from a conjugate heat transfer design campaign for battery cold plates, in three families: matching unlabelled test results to their operating points, choosing which stored runs to reveal before signing off a design, and ranking new duties by peak cell temperature. Every answer is taken from a converged simulation, not from an LLM judge. The best of ten frontier models, GPT-6 Astra, scores 0.71, and models state almost the same confidence on tasks they get wrong as on tasks they get right. We also post-train Qwen3.5-9B with LoRA and reinforcement learning, rewarded by the share of answer criteria the simulation results confirm; on the held-out test split its score rises from 0.071 to 0.106. We release the train, validation and test splits, the evaluation harness and the leaderboard.