GAAC：面向个人设备带宽高效 LLM 推理的无损 KV 压缩
GAAC: Lossless KV Compression for Bandwidth-Efficient LLM Inference on Personal Devices
Yipin Guo · Arun M George · Siddharth Joshi
中文摘要
仅权重的有损量化大幅降低了大型语言模型（LLM）的存储与内存需求，但不断增长的键值（KV）缓存仍是个人设备上长上下文推理的瓶颈。Group-Adaptive Additive Codec（GAAC）是一种无损 KV 压缩与服务系统，利用指数冗余来减少存储并加速注意力计算，且不引入 KV 近似。GAAC 对连续区间使用经校准的逐通道 base-plus-offset 编码来编码 BF16 指数，并在小分组间自适应调整码宽，同时逐位保留 KV 值。融合注意力内核即时重建 KV 分块，避免全缓存解压；块级压缩与固定容量槽位支持增量缓存更新和分页服务。在五个测试模型上，GAAC 实现 1.405-1.407 倍的无损 BF16 KV 压缩；在低精度张量上，FP8 E4M3 缓存为 1.09-1.14 倍，E5M2 缓存为 1.25-1.30 倍。在 NVIDIA DGX Spark 上，64K token 上下文下解码注意力加速 1.19-1.27 倍，端到端 W4A16 解码步骤加速 1.04-1.15 倍；服务场景下常驻 KV 内存减少约 26%，在可比 KV 内存预算下总吞吐提升 13-29%。在 Apple M4 上，GAAC 实现 1.02-1.25 倍注意力加速，证明其可移植性不限于 NVIDIA GPU。
关键要点
- 01长上下文推理中 KV 缓存成为个人设备瓶颈，仅权重量化无法解决
- 02利用 BF16 指数冗余做逐通道无损编码，分组自适应码宽且逐位保真
- 03融合注意力内核即时重建 KV 分块，支持增量更新与分页服务
- 04五个模型上实现 1.405-1.407 倍无损 BF16 KV 压缩
- 05DGX Spark 上解码注意力加速 1.19-1.27 倍，常驻 KV 内存降约 26%
解读
尚无解读。
原始英文摘要
arXiv:2610.38210v1 Announce Type: new Abstract: Weight-only, lossy quantization substantially reduces the storage and memory requirements for serving large language models (LLMs), but the growing key-value (KV) cache remains a bottleneck for long-context inference on personal devices. We present Group-Adaptive Additive Codec (GAAC), a lossless KV compression and serving system that exploits exponent redundancy to reduce storage and accelerate attention without introducing KV approximation. GAAC encodes BF16 exponents using calibrated per-channel base-plus-offset codes over contiguous ranges. GAAC adapts code widths across small groups while preserving KV values bit-for-bit. A fused attention kernel reconstructs KV tiles on the fly, avoiding full-cache decompression. Block-wise compression and fixed-capacity slots support incremental cache updates and paged serving. Across five tested models, GAAC achieves 1.405-1.407$\times$ lossless BF16 KV compression. We also study compression performance on low-precision tensors FP8 E4M3 caches ( 1.09-1.14$\times$) and E5M2 caches (1.25-1.30$\times$). On NVIDIA DGX Spark, GAAC accelerates decode attention at a 64K-token context by 1.19-1.27$\times$ and end-to-end W4A16 decode steps by 1.04-1.15$\times$. In serving, it reduces resident KV memory by approximately 26% and increases aggregate throughput by 13-29% under comparable KV-memory budgets. On Apple M4, GAAC achieves 1.02-1.25$\times$ attention speedups, demonstrating portability beyond NVIDIA GPUs.