TalkFlow:面向语音驱动的整体动作生成的结构化潜在流匹配
TalkFlow: Structured Latent Flow Matching for Speech Driven Holistic Motion Generation
Xinmu Wang · Fumi Honda · Xiang Gao · Xianfeng Gu · Xiyun Song · Heather Yu
中文摘要
语音驱动的整体动作生成旨在从语音合成协调的面部表情、手势和身体动作,其难点在于各部位异构的动态、长程时间连贯性以及跨部位协调。直接建模高维动作可能不稳定,而过于紧凑或无结构的潜在空间可能丢失细粒度动作细节。TalkFlow 是一个用于语音驱动整体动作生成的结构化潜在流框架。它学习上半身动作、下半身动作和面部表情的连续三流潜在表示,保留各部位特有的动态,而不将异构动作成分强行压入单一潜在空间。为保持整体协调,在潜在表示学习和基于流的生成中均引入轻量跨流交互。以多级语音特征为条件,TalkFlow 执行语音条件潜在流匹配,学习从源潜在到真实动作潜在的连续传输过程。在 BEAT2 数据集上的实验在面部表情准确度、身体手势质量和语音-动作同步方面表现强劲。定性比较和用户研究进一步表明,TalkFlow 比现有最先进方法生成更自然、更具表现力且与语音更对齐的整体动作。项目页面:https://xinmuwang.github.io/TalkFlow/。
关键要点
- 01问题:语音驱动整体动作生成面临部位异构动态、长程连贯与跨部位协调难题。
- 02方法:学习上半身、下半身、面部三流连续潜在表示,避免异构动作挤入单一空间。
- 03方法:在潜在表示学习与流生成中引入轻量跨流交互,并以多级语音特征为条件做潜在流匹配。
- 04结果:在 BEAT2 数据集上面部表情准确度、身体手势质量与语音-动作同步表现强劲。
- 05结果:定性比较与用户研究显示生成动作比现有最先进方法更自然、更具表现力且更对齐语音。
解读
尚无解读。
原始英文摘要
arXiv:2610.40972v1 Announce Type: cross Abstract: Holistic co-speech motion generation aims to synthesize coordinated facial expressions, hand gestures, and body movements from speech, which is challenging due to heterogeneous part-specific dynamics, long-range temporal coherence, and cross-part coordination. Directly modeling high-dimensional motion can be unstable, while overly compact or unstructured latent spaces may lose fine-grained motion details. We present \textbf{TalkFlow}, a structured latent flow framework for speech-driven holistic motion generation. TalkFlow learns a continuous three-stream latent representation for upper-body motion, lower-body motion, and facial expressions, preserving part-specific dynamics without forcing heterogeneous motion components into a single latent space. To maintain holistic coordination, lightweight cross-stream interactions are incorporated in both latent representation learning and flow-based generation. Conditioned on multi-level speech features, TalkFlow performs speech-conditioned latent flow matching, learning a continuous transport process from source latents to realistic motion latents. Experiments on the BEAT2 dataset demonstrate strong performance across facial expression accuracy, body gesture quality, and speech-motion synchronization. Qualitative comparisons and user studies further show that TalkFlow generates more natural, expressive, and speech-aligned holistic motion than state-of-the-art methods. The project page is available at https://xinmuwang.github.io/TalkFlow/.