预训练图像编码器的表征风险
Representation Risk in Pretrained Image Encoders
Ardyn Nordstrom · Morgan Nordstrom · Vamuyan Sesay · Matthew D. Webb
中文摘要
应用研究者越来越多地将图像转换为预训练编码器生成的特征,并在下游预测模型中使用这些特征。编码器通常被视为实现细节。但研究表明,编码器实际上可能是模型不确定性的重要来源。这种不确定性被称为表征风险:不同的预训练编码器将同一图像映射到不同的特征空间,可能导致基于预测性能的样本外结论截然不同。研究比较了10种现代和传统冻结编码器,应用于房价、赛马表现、乳腺癌组织学、胸部X光片、连续面部年龄和水稻疾病等场景。在常用维度控制、预测头和安全分组分割下,验证选中SigLIP 2用于房价预测,将测试R²从ResNet50的0.396提升至0.629;选中DINOv2用于赛马,将R²从0.029提升至0.105。没有编码器在所有任务中表现最优。候选程序使用训练数据构建,在独立验证分区上比较。选中程序在房价上达到0.658,肺炎分类准确率达0.978。固定分割的收益在赛马和水稻上较小,而重复分割揭示了赛马特征联合的不稳定性。连续年龄任务选中SigLIP 2,MAE为4.786年。主要表征差距在使用神经预测头、大小相似的DINOv2和ViT模型以及有限适配时仍然存在。这些结果支持一个简单工作流:对可能的表征进行基准测试,在锁定验证数据上选择,仅在有独立验证证据时才合并,并报告配对和分割级不确定性。研究在LOOKAGAIN-ML软件包中实现了这一工作流,用于开展本文分析。
关键要点
- 01不同预训练编码器将同一图像映射到不同特征空间,导致样本外预测结论可能截然不同,称为表征风险
- 02比较10种冻结编码器:房价任务SigLIP 2最优(R² 0.629),赛马任务DINOv2最优(R² 0.105),无编码器全优
- 03验证选中程序在房价达到R² 0.658、肺炎分类准确率0.978、年龄预测MAE 4.786年
- 04赛马和水稻任务固定分割增益小,重复分割揭示特征联合的不稳定性
- 05建议工作流:基准测试表征、在锁定验证数据上选择、有验证证据时才合并、报告不确定性
解读
尚无解读。
原始英文摘要
arXiv:2609.35470v1 Announce Type: cross Abstract: Applied researchers increasingly convert images into features with pretrained encoders, then use those features in a downstream prediction model. The encoder is often treated as an implementation detail. We show that it can instead be a consequential source of model uncertainty. We call this uncertainty representation risk: plausible pretrained encoders map the same images into different feature spaces and can yield sharply different out-of-sample conclusions from predictive performance. We compare ten modern and legacy frozen encoders across applications involving house prices, racehorse performance, breast-cancer histology, chest radiographs, continuous facial age, and rice disease. With common dimension control, heads, and group-safe splits, validation selects SigLIP 2 for houses, raising test $R^2$ from 0.396 for ResNet50 to 0.629, and DINOv2 for horses, raising $R^2$ from 0.029 to 0.105. No encoder is best in every task. Candidate procedures are constructed using training data and compared on a separate validation partition. The selected procedure reaches 0.658 for houses and 0.979 accuracy for pneumonia. Fixed-split gains are small for horses and rice, while repeated partitions reveal instability in horse feature union. Continuous age selects SigLIP 2 at 4.786 years MAE. The principal representation gaps persist with neural heads, similarly sized DINOv2 and ViT models, and limited adaptation. These results support a simple workflow: benchmark plausible representations, select on locked validation data, combine only when separate validation evidence justifies the additional cost, and report paired and split-level uncertainty. We implement this workflow in LOOKAGAIN-ML, the software package used to conduct the analyses in this paper.