跳到正文
返回论文列表
cs.CL提交于 已译

当目标与评估者不一致:评估事后人口统计框架缓解的迁移性

When Objectives and Evaluators Disagree: Evaluating Transfer for Post-hoc Demographic Framing Mitigation

Tunazzina Islam

中文摘要

事后重写用于在源模型不可访问时减少 LLM 生成文本中的人口统计差异。重写可能仅仅因为改善了自身目标或满足了选择器而看似成功。研究将其视为迁移问题,基于 660 个性别对和 1,980 个年龄对比的人口统计条件气候信息。重写器提出编辑,选择模型在满足约束的候选中选择,独立评估者(不参与前两者)对成对框架差异评分。始终针对 Persuasion Bias Index (PBI) 较低信息的固定策略在不同轴和结果聚合上迁移不一致。自适应策略在两侧提出编辑但最多接受一个,在来自两个评估者家族的八项比较中的七项上优于该规则(应用于相同候选),主要靠编辑更多对而非引导方向选择,但保留风险更高。编辑两条信息的策略在更少对上给出最大点降低。这两种策略在保留阈值扫描中减少差异;绝对评分因评估者家族和提示而异。缩小 PBI 差距在两个评估者家族下都增加维度平均差异。缓解应报告为跨目标、评估者及其提示、编辑策略的迁移。

关键要点

  1. 01研究事后重写减少 LLM 文本人口统计差异的迁移问题,使用 660 性别对和 1,980 年龄对比。
  2. 02固定策略针对低 PBI 信息迁移不一致;自适应策略在八项比较中七项更优,但保留风险更高。
  3. 03自适应策略改进主要来自编辑更多对,而非引导方向选择;编辑两条信息策略在更少对上点降低最大。
  4. 04缩小 PBI 差距反而增加维度平均差异;绝对评分因评估者家族和提示而异。
  5. 05建议将缓解报告为跨目标、评估者、提示和编辑策略的迁移。

解读

尚无解读。

原始英文摘要

arXiv:2611.25258v1 Announce Type: cross Abstract: Post-hoc rewriting is used to reduce demographic disparities in LLM-generated text when the source model is inaccessible. A rewrite can look successful merely because it improved its own target or satisfied its selector. We study this as a transfer problem on 660 gender pairs and 1,980 age contrasts of demographic-conditioned climate messages. A rewriter proposes edits, a selection model chooses among constraint-satisfying candidates, and an independent evaluator uninvolved in either rates pairwise framing disparity. A fixed policy that always targets the message with the lower Persuasion Bias Index (PBI) transfers inconsistently across axes and outcome aggregations. An adaptive policy that proposes edits on both sides but accepts at most one significantly improves on that rule, applied to the same candidates, in seven of eight comparisons across two evaluator families, mostly by editing more pairs, not by guided direction choice, at higher preservation risk. A policy editing both messages gives the largest point reductions on fewer pairs. These two policies reduce disparity across a preservation-threshold sweep; absolute ratings vary across evaluator families and prompts. Shrinking the PBI gap increases dimension-mean disparity under both evaluator families. Mitigation should be reported as transfer across objectives, evaluators and their prompts, and editing policies.

同方向论文 · cs.CL

查看全部 →