持续演化 AI 分析师的终身有效决策
Lifetime-valid decisions by continually evolving AI analysts
Xiaojing Sun · Yuhan Zeng · Xiao Wang
中文摘要
持续验证(continual validation)在部署前,使用零假设下的模拟数据评估每个候选 AI 分析师版本,并允许场景随机化和逐步扩展。论文证明,无论更新方式及提案数量如何,部署第一类错误率超限版本的终身概率始终有界,从而确保后续决策在这些场景中将第一类错误率控制在目标水平。第一类错误率足够低的更新则大概率通过验证,使分析师能够持续提升性能且不牺牲严谨性。在生物等效性研究和产品 A/B 测试两个演化示例中,无持续验证时第一类错误率分别达到 0.098 和 0.183;采用该方法后最高仅为 0.045 和 0.044,同时保留最优有效更新功效的 82% 和 96%。
关键要点
- 01演化与反馈自适应可能累积第一类错误并导致过拟合
- 02通过零假设模拟数据部署前验证每个候选版本
- 03终身部署错误率超限版本的概率保持有界
- 04低第一类错误率更新可高概率通过并持续改进
- 05两个示例中第一类错误率均降至不高于 0.045
解读
尚无解读。
原始英文摘要
arXiv:2610.30562v1 Announce Type: cross Abstract: AI agents increasingly act as analysts whose decision can often be cast as hypothesis tests. Classical tests control the Type I error rate, the probability of rejecting a true null hypothesis, for fixed procedures. However, AI analysts introduce internal randomness and can evolve continuously over time. As AI analysts evolve, the risk of approving versions with Type I error rates above the target can accumulate, and adaptive use of evaluation feedback can lead to overfitting. We introduce continual validation, which evaluates each proposed AI analyst version before deployment using simulated data under the null hypothesis, with scenarios that can be randomized and expanded over time. We prove that, regardless of how updates are chosen or how many are proposed, the lifetime probability of ever deploying a version whose Type I error rate exceeds the allowed level remains bounded. This guarantees that every subsequent decision controls the Type I error rate at the target level across these scenarios. At the same time, updates with sufficiently low Type I error rates pass validation with high probability, allowing the analyst to continuously improve its performance without sacrificing rigor. In two examples involving evolving AI analysts for bioequivalence studies and product A/B tests, Type I error rates reached 0.098 and 0.183 without continual validation, compared with at most 0.045 and 0.044 with it, while retaining 82% and 96% of the power from optimally selected valid updates.