一条曲线,不同权重:AUC、Brier 分数、决策加权 AUC 与策略 Youden 指数
One Curve, Different Weights: AUC, the Brier Score, Decision-Weighted AUC, and the Policy Youden Index
Yegor Baranovski
中文摘要
AUC 与 Brier 分数是二分类风险评分的标准汇总指标,但二者对模型的排序可能不同。引入加权 AUC(AUC_π),按概率为 π 的决策者对不同观测对的处理差异对每对观测加权。证明三者均基于同一条曲线——Youden 指数 J(t)=TPR(t)-FPR(t) 在截断点 t 上的加权平均,区别仅在截断点权重:AUC 用全体分数分布加权,Brier 分数用真实概率 E[Y|S] 加权,AUC_π 用决策者行动概率加权。归一化后,AUC_π 等于理想预测的 Brier 技能分数,其权重移至决策者关心的截断点。进一步证明,按 |π(s)-π(s')| 对观测对加权(至多相差尺度)是唯一使成对加权 AUC 对任意分数分布都等于 Youden 曲线平均的预设成对权重。作为排序指标,AUC 对所有模型一视同仁;Hand (2009) 表明它也可读作成本阈值上的平均,但此读法下各模型对阈值加权不同。AUC_π 的分子称为策略 Youden 指数,等于固定策略下期望作用于类 1 的比例减去类 0 的比例,以同一策略评判所有模型从而避免上述问题;完整比值 AUC_π 则不然。
关键要点
- 01AUC、Brier 分数与 AUC_π 均为 Youden 曲线 J(t)=TPR(t)-FPR(t) 的加权平均,仅截断点权重不同。
- 02AUC_π 按决策者行动概率加权,归一化后等于理想预测的 Brier 技能分数。
- 03|π(s)-π(s')| 是唯一使成对加权 AUC 对任意分数分布都等于 Youden 曲线平均的预设成对权重。
- 04AUC 作为排序指标对所有模型权重相同,但按成本阈值读法时各模型对阈值加权不同。
- 05策略 Youden 指数以同一策略评判所有模型,避免 AUC_π 完整比值带来的模型间权重差异。
解读
尚无解读。
原始英文摘要
arXiv:2611.14465v1 Announce Type: cross Abstract: AUC and the Brier score are standard summaries of a binary risk score, yet they can rank models differently. We introduce a weighted AUC, AUC$_\pi$, which weights each pair of observations by how differently a decision maker with probability of acting $\pi$ treats them. We show that all three are built on weighted averages of one curve, the Youden index $J(t) = \mathrm{TPR}(t) - \mathrm{FPR}(t)$ across cutoffs $t$, and that the averages differ in the weights they put on the cutoffs. Each weights a cutoff by how much something rises across it: for AUC, the distribution of all scores; for the Brier score, the true probability $\mathbb{E}[Y \mid S]$; for AUC$_\pi$, the decision maker's probability of acting. Up to normalization, AUC$_\pi$ is the Brier skill score of the ideal forecast with its weight moved to the cutoffs the decision maker cares about. We also show that weighting pairs by $|\pi(s) - \pi(s')|$ is, up to scale, the only pair weight fixed in advance that makes a pair-weighted AUC an average of the Youden curve for every score distribution. As a ranking measure, AUC weights every model the same way. Hand (2009) shows that it can also be read as an average over cost thresholds, and in that reading each model weights the thresholds differently. The numerator of AUC$_\pi$, which we call the policy Youden index, is the expected share of class 1 a fixed policy acts on minus the expected share of class 0. It judges every model by the same policy and so avoids this; the full ratio AUC$_\pi$ does not.