ViBES T2M

ViBES-T2M (Qwen-MoME): frozen-LLM text-to-motion at specialist-SOTA quality

Text-to-motion model in the ViBES AR-MoME (mixture-of-modality-experts) architecture: a fully frozen Qwen3-1.7B text backbone plus a narrow trainable motion expert (~0.27B). The text LLM is untouched — a visibility rule (text attends only to text; motion attends causally to text+motion) makes freezing exact, so the base model's language ability is fully preserved.

Results (HumanML3D test, Guo et al. protocol, 20 replications)

Model	Params (trainable)	Frozen text LLM	R@1	R@3	FID
T2M-GPT	full	no	0.492	0.679	0.141
MoMask	full	no	0.521	0.807	0.045
ViBES-T2M (this)	0.27B	yes	0.489	0.769	0.265

R@1 on par with T2M-GPT while keeping the entire conversational LLM intact.

Demos

GT | ViBES-T2M (frozen 1.7B) | ViBES-9B:

"a man walks forward, then squats to pick something up with both hands, stands back up, and resumes walking"

demo1

"the man runs backwards"

demo2

Files

• vibes_t2m_qwen_mome_1p7b.pt — motion-expert weights (mot_state), training config, metrics. The frozen base is downloaded from Qwen/Qwen3-1.7B at load time.
• modeling_qwen_mome.py — self-contained model definition (needs transformers with Qwen3).

Motion tokenizer

MotionGPT's public HumanML3D VQ-VAE (motiongpt_s3_h3d.tar, 512 codes, 4x temporal downsample, 263-D HumanML3D features). Motion vocab = 514 (512 codes + BOS/EOS). Decode generated codes with that VQ-VAE + the Comp_v6_KLD01 mean/std.

Usage

import torch
from modeling_qwen_mome import QwenMoMEForT2M

ck = torch.load("vibes_t2m_qwen_mome_1p7b.pt", map_location="cpu", weights_only=False)
model = QwenMoMEForT2M("Qwen/Qwen3-1.7B", d_mot=512, ffn_mot=4096, mot_vocab=514)
model.load_state_dict(ck["mot_state"], strict=False)   # base.* comes from the HF download
model = model.cuda().eval()

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
text = "Give me a gesture that corresponds to a person walks forward then squats down"
ids = torch.tensor([tok(text, add_special_tokens=False)["input_ids"]]).cuda()
codes = model.generate_motion(ids, max_new=60, greedy=False,
                              temperature=0.7, top_p=0.9, cfg_scale=2.5)
# codes: MotionGPT-512 VQ indices -> decode with motiongpt_s3_h3d.tar to 263-D motion

Recommended decoding: sampling temperature=0.7, top_p=0.9 with classifier-free guidance cfg_scale 2.5–3 (the model was trained with 10% caption dropout).

Training recipe (for reproduction)

Frozen Qwen3-1.7B; per-layer motion expert (q/k/v/o + q/k-norm + RMSNorms + SwiGLU 512→4096), own 514-token embedding and untied head; plain-arange positions; motion-only shifted CE; bs 16, lr 2e-4 cosine, wd 0.05, clip 1.0, caption-dropout 0.1; peak at ~10k steps (select checkpoints by generation quality on the official val split, not teacher-forced loss). Plain PyTorch bf16 (no DeepSpeed).

License

Research use only. The frozen backbone follows the Qwen3 license; the motion tokenizer follows MotionGPT's license; HumanML3D data terms apply.

Part of the ViBES project (speech-language-behavior modeling).