--- license: apache-2.0 base_model: Wan-AI/Wan2.1-T2V-14B tags: - lip-sync - talking-head - video-to-video - video-generation - diffusion - autoregressive - distillation - wan2.1 --- # Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization [Paper (arXiv:2606.11180)](https://arxiv.org/abs/2606.11180) · [Project page](https://cvlab-kaist.github.io/LipForcing/) · [Code (GitHub)](https://github.com/cvlab-kaist/LipForcing) **Lip Forcing** is, to our knowledge, the first autoregressive diffusion method for video-to-video (V2V) lip synchronization. It distills a high-fidelity bidirectional 14B audio-conditioned teacher (*OmniAvatar-LS*, our lip-sync finetune of OmniAvatar) into causal few-step students that generate each chunk in just **two denoising steps with no inference-time CFG** - enabling real-time, streaming lip-sync on a reference video with sub-millisecond time-to-first-frame. ## Files in this repository | File | What | For | |------|------|-----| | `lipforcing_14b.pth` | 14B student - merged, self-contained (base Wan + OmniAvatar-LS adapter + Self-Forcing student baked into one file) | inference | | `teacher/omniavatar_ls_14b.pt` | OmniAvatar-LS 14B V2V adapter (frozen distillation teacher + student init) | training | | *1.3B student* | *coming soon (same repo)* | | ## Quickstart Install the code from [GitHub](https://github.com/cvlab-kaist/LipForcing): ```bash git clone https://github.com/cvlab-kaist/LipForcing.git && cd LipForcing conda create -y -n lipforcing python=3.12 && conda activate lipforcing pip install -e . ``` Download this checkpoint plus the small external encoders/decoders (commands for every component are in the repo README's *Weights* section), then: ```bash python scripts/inference/inference_streaming.py \ --ckpt_path weights/lipforcing_14b.pth \ --vae_path weights/Wan2.1-T2V-14B/Wan2.1_VAE.pth \ --wav2vec_path weights/wav2vec2-base-960h \ --mask_path weights/mask.png \ --taehv_ckpt weights/taew2_1.pth \ --text_encoder_path weights/Wan2.1-T2V-14B/models_t5_umt5-xxl-enc-bf16.pth \ --video_path ref.mp4 --audio_path speech.wav --output_path out.mp4 ``` Streaming inference encodes, denoises, decodes, and composites each AR chunk on the fly - first frames arrive before the clip finishes, GPU memory stays constant for any clip length (**~37 GB** peak at 14B with precomputed text embeddings; ~50 GB with runtime T5 encoding), and face detection + 512×512 alignment + paste-back run automatically, so any talking-head video works as input. Training (two stages: Diffusion-Forcing init, then Self-Forcing DMD distillation with the SyncNet reward) is documented in the repo's [README](https://github.com/cvlab-kaist/LipForcing#training) and [DATA.md](https://github.com/cvlab-kaist/LipForcing/blob/master/DATA.md). ## Acknowledgements Builds on [NVIDIA FastGen](https://github.com/NVlabs/FastGen), [Self Forcing](https://github.com/guandeh17/Self-Forcing), [Reward-Forcing](https://github.com/JaydenLyh/Reward-Forcing), [OmniAvatar](https://github.com/Omni-Avatar/OmniAvatar), [LatentSync](https://github.com/bytedance/LatentSync), and [Wan2.1](https://github.com/Wan-Video/Wan2.1). Third-party licenses ship in the [code repository](https://github.com/cvlab-kaist/LipForcing/tree/master/licenses). ## Citation ```bibtex @article{cho2026lipforcing, title = {Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization}, author = {Cho, Paul Hyunbin and Jang, Jinhyuk and Lee, SeokYoung and Lee, Joungbin and Jin, Siyoon and Shin, Heeseong and Yi, Jung and Park, Yunjin and Park, Chulmin and Kim, Seungryong}, journal = {arXiv preprint arXiv:2606.11180}, year = {2026}, } ```
