跳到正文
返回论文列表
stat.ME提交于 已译

并非所有错误标签都相同:基于置信序列的任务感知高效审计

Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences

Fangyuan Lin · Noah Krever · Jakub \v{C}ern\'y · Lily Xu

中文摘要

研究固定数据集中类别标签的高效自适应审计,动机来自生物多样性信用项目:物种记录专家核验成本高,下游评分(如付款)对某些标签错误更敏感。提出的 ATLAS 框架将任意时间有效推断与任务感知自适应审计策略结合。通过混淆矩阵表示标签修正,依据标签不确定性和潜在修正对下游评分函数的影响,优先选择下一个检查的数据点。在肯尼亚生物多样性数据的主要案例中,任务感知审计在保持任意时间有效保证的同时,减少了估计错误标签率和非线性生物多样性评分所需的专家审查量。还在图像分类和命名实体识别的公开数据集上评估 ATLAS。在线性和非线性评分的实验中,ATLAS 通常以显著更少的审计达到与竞争任意时间有效方法相同的精度。当无分布保证成本过高时,开发基于模型的贝叶斯替代方案,依赖建模假设进一步减少审计次数。生物多样性案例研究突显统计有效性与实际可行性之间的差距:Serfling 界和现有置信序列程序等通用方法严谨,但实践中可能需要过多专家审查。结果表明,任务感知审计能显著减少所需专家审计次数,同时提供有限样本、任意时间有效的保证。

关键要点

  1. 01问题:通用任意时间有效方法在专家核验成本高时审计次数过多,统计有效性与实际可行性存在差距。
  2. 02方法:ATLAS 结合任意时间有效推断与任务感知自适应策略,按标签不确定性和对下游评分的影响优先审计。
  3. 03结果:在肯尼亚生物多样性数据中减少专家审查量,在图像分类和 NER 上以更少审计达到相同精度。
  4. 04方法:无分布保证成本过高时,提供基于模型的贝叶斯替代方案以进一步减少审计次数。
  5. 05结果:任务感知审计在保持有限样本、任意时间有效保证的同时显著降低专家审计需求。

解读

尚无解读。

原始英文摘要

arXiv:2610.33262v2 Announce Type: replace-cross Abstract: We study efficient, adaptive auditing of categorical labels in a fixed dataset, motivated by biodiversity credit programs in which expert verification of species records is costly and downstream scores (e.g., payment) are more sensitive to errors on certain labels than on others. Our proposed ATLAS framework couples anytime-valid inference with a task-aware adaptive auditing policy. Representing label corrections through a confusion matrix, we prioritize which data points to inspect next based on both label uncertainty and the influence of potential label corrections on the downstream score function. In our main case study on Kenyan biodiversity data, task-aware auditing reduces the expert review needed to estimate both the mislabel rate and the nonlinear biodiversity score while maintaining anytime-valid guarantees. We also evaluate ATLAS on public datasets for image classification and named-entity recognition. Across our experiments with linear and nonlinear scores, ATLAS generally reaches the same precision as competing anytime-valid methods with substantially fewer audits. When distribution-free guarantees are prohibitively costly, we develop a model-based Bayesian alternative that can further reduce the number of audits by relying on modeling assumptions. Our biodiversity case study highlights a gap between statistical validity and practical feasibility: generic methods such as Serfling's bound and existing confidence-sequence procedures are rigorous but can require prohibitively many expert reviews in practice. Together, these results show task-aware auditing can significantly reduce the number of expert audits needed while providing finite-sample, anytime-valid guarantees

同方向论文 · stat.ME

查看全部 →