metalSPARSE:面向 Apple Silicon GPU 的 2:4 半结构化稀疏 GEMV 内核
metalSPARSE: A 2:4 Semi-Structured Sparse GEMV Kernel for Apple Silicon GPUs
Tek Raj Awasthi (Friedrich-Alexander-Universit\"at Erlangen-N\"urnberg) · Babita Awasthi
中文摘要
自回归大语言模型(LLM)逐 token 生成文本,每个 token 都要从内存读取全部权重,因此在边缘设备上解码受内存带宽限制。2:4 半结构化稀疏保留每四个权重中的两个,但 Apple Silicon GPU 没有稀疏矩阵硬件,只有内核真正读取更少字节时剪枝才有收益。metalSPARSE 是据我们所知首个面向 Apple GPU 的 2:4 稀疏内核,并配有 Metal 编写的融合解码器。内核以压缩形式流式读取权重,并在寄存器中展开。在 Qwen2.5-1.5B 上对所有解码器线性层施加 2:4 稀疏后,在 8 GB M2 MacBook Air 上解码比 PyTorch-MPS 上的 HuggingFace Transformers 快 3.6 倍,在 16 GB M4 上比 MLX 快 1.52 倍。同一引擎在稠密权重上与 MLX 持平,说明加速来自压缩权重,稀疏化实现了其字节缩减所允许加速的 97%。计算密集的提示处理没有加速。为保持质量,将 Wanda 掩码与分块重建及低秩适配器结合,适配器与保留权重联合训练并量化为 4 比特。在 WikiText-2 上,困惑度从一次性剪枝(Wanda 加闭式重拟合)后的稠密模型 3.01 倍降至 1.34 倍,整个流程在同一笔记本硬件上运行。
关键要点
- 01问题:Apple Silicon GPU 无稀疏硬件,2:4 剪枝需内核真正减少内存读取才有收益。
- 02方法:首个 Apple GPU 2:4 稀疏 GEMV 内核,压缩流式读取权重并在寄存器展开,配 Metal 融合解码器。
- 03结果:Qwen2.5-1.5B 上比 HuggingFace/PyTorch-MPS 快 3.6 倍,比 MLX 快 1.52 倍,达字节缩减允许加速的 97%。
- 04局限:计算密集的提示处理不加速;稠密权重下与 MLX 持平。
- 05质量保持:Wanda 掩码加分块重建与 4 比特低秩适配器,WikiText-2 困惑度从 3.01 倍降至 1.34 倍。
解读
尚无解读。
原始英文摘要
arXiv:2611.14545v1 Announce Type: cross Abstract: Autoregressive Large Language Models (LLMs) generate text one token at a time, and each token reads every weight from memory, so on edge devices, decoding is limited by memory bandwidth. 2:4 semi-structured sparsity keeps two of every four weights, but Apple-silicon GPUs have no sparse matrix hardware, so pruning helps only if a kernel actually reads fewer bytes. We present metalSPARSE, to our knowledge the first 2:4 sparse kernels for Apple GPUs, together with a fused decoder written in Metal. The kernels stream weights in compressed form and expand them in registers. On Qwen2.5-1.5B with 2:4 sparsity on every decoder linear layer, decoding runs 3.6x faster than HuggingFace Transformers on PyTorch-MPS on an 8 GB M2 MacBook Air and 1.52x faster than MLX on a 16 GB M4. The same engine on dense weights matches MLX, so the gain comes from the compressed weights, and sparsity delivers 97% of the speedup its byte reduction allows. Prompt processing, which is compute-bound, does not speed up. To preserve quality, we combine a Wanda mask with block-wise reconstruction and low-rank adapters trained jointly with the surviving weights and quantized to 4 bits. On WikiText-2 this brings perplexity from 3.01x dense after one-shot pruning (Wanda with a closed-form refit) to 1.34x, and the whole pipeline runs on the same laptop/hardware.