Unreal Audio2lipsync

--- license: mit tags: - lipsync - audio-to-face - metahuman - arkit - blendshapes - unreal-engine - livelink library_name: pytorch pipeline_tag: audio-to-audio --- # Audio2Lipsync Audio-driven lipsync model for Unreal Engine MetaHumans. Given a mono 16 kHz audio clip, the model predicts 52-channel ARKit blendshape coefficients at 60 fps — frame-accurate mouth, jaw, tongue, and cheek motion that drops straight into the MetaHuman face rig via LiveLink. Companion code repository: https://github.com/aaryansachdeva/unreal-audio2lipsync ## Architecture - **Encoder:** HuBERT-Large (frozen) — phonetic features from 16 kHz audio - **Decoder:** 8-layer Transformer (d=512, 8 heads, FF=2048, dropout=0.2) - **Output:** 52 ARKit blendshape coefficients per frame at 60 fps - **Loss:** Weighted masked L1 (mouth channels get 3× weight) + velocity loss - **Training data:** iPhone Live Link Face recordings (ARKit blendshapes) paired with their audio tracks ## Files | File | Purpose | |---|---| | `best.pt` | EMA inference checkpoint (trainable params only; frozen HuBERT loaded from `torchaudio`) | ## Usage Install the code repo and point the sidecar server at this checkpoint: ```bash git clone https://github.com/aaryansachdeva/unreal-audio2lipsync.git cd unreal-audio2lipsync pip install -r python/requirements.txt # Download best.pt from this HF repo into a local checkpoints/ dir, then: python python/src/server.py --ckpt checkpoints/best.pt --stats-dir python/stats ``` See the [code repo README](https://github.com/aaryansachdeva/unreal-audio2lipsync) for full Unreal Engine integration instructions. ## License MIT — use commercially, modify, redistribute, no obligations beyond keeping the license notice. See [LICENSE](https://github.com/aaryansachdeva/unreal-audio2lipsync/blob/master/LICENSE) in the code repo.

Source context: 0 downloads · 2 likes · Pipeline audio-to-audio · Library pytorch · Repo fotonlabs/unreal-audio2lipsync