FedPA-LoRA：面向异构联邦 LoRA 中聚合与初始化误差缓解的乘积对齐框架
FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
Juseok Jeon · Ramy E. Ali · Doyun Kwon · Myungbeom Her · Jinhwi Kim · Jinhyun So
中文摘要
低秩适配（LoRA）支持大语言模型的高效联邦微调，但其分解结构在聚合保真度与本地优化连续性之间形成张力。按因子聚合会带来聚合失配，但能更好地保持因子连续性；乘积空间重建可降低这种失配，代价是更大的因子级初始化失配。FedPA-LoRA 是一个乘积对齐框架，同时处理这两类失配，并为同构与异构秩建立收敛界。客户端跨轮保留本地因子以消除初始化失配，并将本地乘积与按秩区分的全局参考对齐以减少客户端漂移。服务器在乘积空间聚合异构秩更新以缓解聚合失配，并在不形成稠密聚合结果的情况下高效重建秩约束的全局适配器。通过解耦本地秩与参考秩，FedPA-LoRA 支持客户端各自的计算与通信预算。在自然语言理解与生成任务上的实验表明，在数据与资源异构条件下均取得稳定增益，其中异构秩下 GLUE 平均准确率提升 6.82 个百分点。代码见 https://github.com/Juseok-Jeon/FedPA-LoRA。
关键要点
- 01问题：LoRA 分解使联邦微调中聚合保真度与本地优化连续性难以兼顾，按因子聚合与乘积空间重建各引入失配。
- 02方法：客户端跨轮保留本地因子并对齐按秩区分的全局参考，服务器在乘积空间聚合异构秩更新并重建秩约束适配器。
- 03结果：同构与异构秩下均给出收敛界，解耦本地秩与参考秩以支持客户端差异化计算与通信预算。
- 04结果：自然语言理解与生成任务在数据与资源异构下稳定增益，异构秩下 GLUE 平均准确率提升 6.82 个百分点。
解读
尚无解读。
原始英文摘要
arXiv:2608.15381v2 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorization creates a tension between aggregation fidelity and local optimization continuity. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch. We propose FedPA-LoRA, a product-aligned framework that addresses both mismatches and establishes convergence bounds for homogeneous and heterogeneous ranks. Clients preserve local factors across rounds to eliminate initialization mismatch and align their products with rank-specific global references to reduce client drift. The server aggregates heterogeneous-rank updates in product space to mitigate aggregation mismatch and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. By decoupling local and reference ranks, FedPA-LoRA supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show consistent gains under data and resource heterogeneity, including a 6.82 percentage-point improvement in average GLUE accuracy under heterogeneous ranks. The code is available at https://github.com/Juseok-Jeon/FedPA-LoRA.