mmPencil: 基于毫米波雷达与大视觉语言模型的书写风格无关空中手写识别
mmPencil: Toward Writing-Style-Independent In-Air Handwriting Recognition via mmWave Radar and Large Vision-Language Model
Yifan Guo · Zhu Wang · Qian Qin · Yangqian Lei · Qiwen Gan · Zhuo Sun · et al.
中文摘要
基于射频信号的空中手写识别虽具潜力，但现有方法受限于预设书写平面、风格约束及小词汇量。m^2VLMs 架构通过模态映射连接毫米波雷达与大视觉语言模型（VLM），构建了 mmPencil 3D 空中手写单词识别系统。该系统利用多阶段空间轨迹重建算法从毫米波信号恢复 3D 轨迹，并通过空间到视觉映射算法转化为视觉表征，降低对书写风格及环境因素的敏感性。研究发布包含 12 名用户、22 种场景、200 个单词共 7,664 个样本的 3D 毫米波手写数据集。实验表明，mmPencil 在位置、距离、角度、速度、大小、方向及用户移动变化下均表现稳健，已知用户 200 类单词准确率达 97.60%，未知用户 50 类单词零样本准确率为 92.50%，优于最先进基线。
关键要点
- 01现有射频手写识别受限于预设平面、风格约束及小词汇量，缺乏通用性。
- 02提出 m^2VLMs 架构，将毫米波信号重建的 3D 轨迹映射为视觉表征供 VLM 识别。
- 03发布包含 7,664 个样本、200 个单词及 12 名用户的 3D 毫米波手写数据集。
- 04已知用户 200 类单词准确率达 97.60%，未知用户 50 类零样本准确率为 92.50%。
解读
尚无解读。
原始英文摘要
arXiv:2610.23419v1 Announce Type: cross Abstract: In-air handwriting recognition based on radio frequency (RF) signals offers a promising solution for human-computer interaction, yet existing methods are often limited by predefined writing planes, constrained writing styles, and small recognition vocabularies. To address these limitations, we propose m^2VLMs, a modality mapping architecture that bridges millimeter-wave (mmWave) radar sensing and large vision-language models (VLMs), and develop mmPencil, a 3D in-air handwriting word recognition system. mmPencil reconstructs 3D handwriting trajectories from mmWave signals using a multi-stage spatial trajectory reconstruction algorithm and maps them into visual representations through a spatial-to-visual mapping algorithm, enabling VLMs to recognize handwriting while reducing sensitivity to writing styles and environmental factors. We collect and release a 3D mmWave handwriting dataset containing 7,664 samples of 200 words from 12 users across 22 writing scenarios. Extensive experiments demonstrate that mmPencil achieves robust word recognition under variations in writing position, range, angle, speed, size, direction, and user movement. It achieves 97.60% accuracy on 200 word classes for seen users and 92.50% zero-shot accuracy on 50 word classes for unseen users, outperforming state-of-the-art baselines.