cs.CL提交于 2026-10-08已译

MIRROR:从模仿到内化的 LLM 个性化

MIRROR: From Imitation to Internalization in LLM Personalization

Huayi Lai · Jicheng Yang · Min Yi · Chong Meng

中文摘要

针对个性化 LLM(大语言模型)的需求正从风格模仿转向内容质量,本研究探讨自蒸馏(self-distillation)是否能在现有微调范式中弥合这一差距。为此提出 MIRROR(Meta-personalization by Internalizing Reference-Revealed On-policy Reflections),一种将 LLM 个性化从模仿转向偏好内化的新型自蒸馏框架。首先,用参考揭示的在线策略自蒸馏取代参考词元(token)模仿,沿模型自身生成轨迹对其下一词元分布与参考条件下的自身分布进行对齐,从而内化用户偏好而非复现参考措辞。其次,引入 MIRROR-F 这一即插即用的聚焦模块,在在线策略分布对齐基础上对信息性参考词元施加选择性监督,在保留用户特有表达的同时强化内容生成。在三个个性化生成基准、两种模型规模以及参考式与 LLM 式互补评估下,MIRROR 与 MIRROR-F 在整体个性化性能与文本质量上均取得领先表现,且在三项未见过的个性化生成任务上相比基于 SFT(监督微调)的基线表现出更少的灾难性遗忘。改进效果在模型规模与应用场景上保持一致,并转化为 LLM 个性化任务上的性能提升。

关键要点

01问题:LLM 个性化需求从风格模仿转向内容质量,现有参考词元模仿微调范式难以兼顾偏好内化与高质量生成。
02方法:MIRROR 用参考揭示的在线策略自蒸馏对齐模型自身生成轨迹上的分布,以内化偏好;MIRROR-F 进一步对信息性参考词元施加选择性监督。
03结果:在三个个性化生成基准与两种模型规模下,整体个性化性能与文本质量领先,且在三个未见任务上灾难性遗忘少于 SFT 基线。
04效果:改进在不同模型规模与应用场景中保持一致,稳定转化为 LLM 个性化任务的实际性能提升。

解读

尚无解读。

原始英文摘要

arXiv:2610.09795v1 Announce Type: new Abstract: The demand for personalized LLMs is shifting from style imitation toward content quality. We investigate whether self-distillation can bridge this gap in existing fine-tuning paradigm. To address this limitation, we introduce MIRROR(Meta- personalization by Internalizing Reference-Revealed On-policy Reflections), a novel self-distillation framework that shifts LLM personalization from imitation toward preference internalization. First, we replace reference-token imitation with reference-revealed on-policy self-distillation, aligning the model's next-token distributions along its own generation trajectories with those of its reference-conditioned self, thereby internalizing user preferences rather than reproducing reference wording.Second, we introduce MIRROR-F, a focal plug-in that augments on-policy distributional alignment with selective supervision over informative reference tokens, thereby strengthening content generation while preserving user-specific expression. Across three personalized generation benchmarks, two model scales, and complementary reference-based and LLM-based evaluations, MIRROR and MIRROR-F achieve leading overall personalization performance and superior text quality, while exhibiting less catastrophic forgetting than SFT-based baselines on three unseen personalized generation tasks. The gains are consistent across model scales and application scenarios, translating to improved performance in LLM personalization tasks.