LoM A2m 4part

Language of Motion — Decoder-only 4-part Audio-to-Motion (a2m)

Speech-driven, whole-body co-speech motion generation as next-token prediction. A pretrained LLM backbone (Qwen3) is turned into an audio→motion model by keeping its text vocabulary and enlarging it with GLM-4-Voice audio tokens and four motion codebooks, then generating the motion compositionally in a single causal stream:

[BOS] audio-tokens [SEP] [FACE] face [UPPER] upper [LOWER] lower [HAND] hand [EOS]

Each motion stream is decoded by its VQ decoder to SMPL-X (face = 6D head + 6D jaw + 100 expr; body = upper / lower / hand), giving a full talking + gesturing avatar from speech alone.

Checkpoints

File	Backbone	Params	val loss ↓	val acc ↑
qwen3-1.7B_4part.pt	Qwen3-1.7B	1.76 B	0.696	0.854
qwen3-0.6B_4part.pt	Qwen3-0.6B	0.61 B	0.705	0.834

(val loss = mean cross-entropy over the 4 motion streams on a held-out split; the two numbers are comparable to each other since both use the same vocab/targets.)

Each .pt is self-describing: {model, cfg, vocab, step, val_loss, val_acc} (torch.load).

Architecture

• Backbone: pretrained Qwen3-1.7B / Qwen3-0.6B (decoder-only). The text vocab is kept; the embedding table is resized (mean-init) to append audio + motion + delimiter tokens.
• Audio tokens: GLM-4-Voice (16384-way, 12.5 tok/s).
• Motion codebooks: ViBES face VQ (512-way, 25 tok/s, 112-D) + upper / lower / hand body VQs (256-way each, 6.25 tok/s).
• Generation: block-by-block (face → upper → lower → hand), each block constrained to its own code range; the decoded VQ streams are composited into a single SMPL-X sequence.

Training data

Trained on a mix of co-speech-gesture and talking-head corpora:

Dataset	Provides	Notes
BEAT2	body + face	CC BY-NC 4.0 (non-commercial)
TFHP (HDTF-derived)	face only (body masked)	talking-head
YouTube_Talking	face only (body masked)	in-the-wild talking-head

For the YouTube_Talking data preparation, follow ViBES. We do not redistribute any raw video/audio — YouTube_Talking is released recipe-only (URL list + downloader + derived annotations) as part of ViBES: JuzeZhang/YouTube_Talking — see its README / the ViBES docs/1-data/youtube_talking.md for the download + annotation recipe. The audio tokenization and motion-token extraction used here are the same scripts described there.

Usage / inference

These weights need the inference harness shipped alongside them (a2f_decoder/) plus the VQ decoders and the GLM-4-Voice tokenizer. Minimal flow:

# 1. audio-tokens -> 4 motion streams -> SMPL-X
python a2f_decoder/infer_4part.py --ckpt qwen3-1.7B_4part.pt --out out/ --save_only
# 2. render (SMPL-X body via Blender + FLAME face)

See a2f_decoder/README.md (included) for the full dependency list (VQ checkpoints, SMPL-X / FLAME model files, GLM-4-Voice tokenizer) and end-to-end example.

License

Research use only. These weights are derived from models trained on BEAT2 (CC BY-NC 4.0, non-commercial), TFHP (HDTF-derived), and YouTube_Talking (source videos remain under YouTube's terms; none are redistributed). The base models are Qwen3 (Apache-2.0). Use of these checkpoints is restricted to non-commercial research and inherits the terms of the underlying datasets and base models. See LICENSE.

Citation

@misc{lom_a2m_4part,
  title  = {Language of Motion: Decoder-only 4-part Audio-to-Motion},
  author = {Zhang, Juze and others},
  year   = {2026},
  note   = {Research-use-only checkpoints, https://huggingface.co/JuzeZhang}
}