内存带宽而非 FLOPs：边缘 GPU 上基于查询的分割解码器剖析与加速
Memory Bandwidth, Not FLOPs: Profiling and Accelerating Query-Based Segmentation Decoders on Edge GPUs
Oliver Busemann (Osnabr\"uck University · CLAAS E-Systems GmbH) · Torben T\"oniges (CLAAS E-Systems GmbH) · Martin Atzmueller (Semantic Information Systems Group · Osnabr\"uck University · German Research Center for Artificial Intelligence)
中文摘要
基于查询的分割模型在工作站 GPU 上实时推理表现优异，但在边缘设备上，其 Transformer 解码器受限于内存带宽，该瓶颈在标准 FLOP 分析中不可见，且随输入分辨率增加和 DRAM 带宽限制而恶化。FP16 屋顶线模型分析显示，FastInst 和 Mask2Former 的非张量核心部分仅达到 CUDA 核心脊线的 2.5% 至 6%，却占据解码器 53% 至 70% 的时间及 61% 至 71% 的字节数。最大内核可持续达到 90% 的标称峰值 DRAM 带宽。瓶颈主要源于跨内核激活流量而非低 GEMM 强度。CompactInst 采用无参数 4 倍空间池化减少解码器内移动字节数，并通过双卷积细化头恢复细节。在 Jetson Orin Nano 上，解码器加速 2.35 至 4.74 倍，模型加速 1.57 至 2.62 倍，精度损失极小（COCO val2017 AP -0.4，Cityscapes test AP +0.2）。
关键要点
- 01边缘设备上 Transformer 解码器受内存带宽限制，标准 FLOP 分析无法揭示此瓶颈。
- 02非张量核心操作虽计算强度低，但占据解码器大部分时间和内存流量，导致带宽饱和。
- 03CompactInst 通过空间池化减少数据移动，在 Jetson Orin Nano 上实现显著加速且精度基本持平。
- 04实验证实解码器延迟由内存字节数而非 FLOPs 主导，即使增加 FLOPs 也能提升速度。
解读
尚无解读。
原始英文摘要
arXiv:2610.22377v1 Announce Type: cross Abstract: Query-based segmentation models achieve strong performance even under real-time inference on workstation GPUs. On edge devices, however, we show their transformer decoders are deeply memory-bandwidth-bound, a bottleneck invisible to standard FLOP analysis that worsens with input resolution and limited DRAM bandwidth. An FP16 roofline analysis with tensor-core GEMMs active and hardware DRAM counters shows that the non-tensor-core aggregates of FastInst and Mask2Former reach only $2.5\%$ to $6\%$ of the CUDA-core ridge yet account for $53\%$ to $70\%$ of decoder time and $61\%$ to $71\%$ of decoder bytes. The largest individual kernels sustain up to $90\%$ of nominal peak DRAM bandwidth, effectively saturating the memory bus. The bottleneck is therefore not primarily low GEMM intensity, but cross-kernel activation traffic. CompactInst addresses this bottleneck using parameter-free spatial pooling at $4{\times}$, which reduces the bytes moved within the transformer decoder, while a two-convolution refinement head recovers spatial detail. Under TensorRT FP16 on Jetson Orin Nano, CompactInst delivers decoder speedups from $2.35{\times}$ to $4.74{\times}$ and model speedups from $1.57{\times}$ to $2.62{\times}$ as input resolution grows, while closely matching accuracy ($-0.4$ AP on COCO val2017, $+0.2$ AP on Cityscapes test). A heavier variant with $2.57{\times}$ more decoder FLOPs still runs its decoder $1.74{\times}$ faster on the most bandwidth-constrained device, confirming at the deployment level that decoder bytes, not FLOPs, govern latency.