JoyAI Echo

JoyAI-Echo 🎬 Pushing the Frontier of Long Video Generation Official model weights for minute-level multi-shot audio-video generation with a distilled DMD generator, paired cross-modal memory, and story-level consistency. For academic research and non-commercial use only. 📄 Paper | 🌐 Project Page | 💻 Inference Code | 🧬 Model | 🚀 Usage | 📊 Results | 📝 Citation ## Model Summary is a long-form, multi-shot, audio-video generation framework that breaks the barriers of error accumulation, weak temporal coherence, and prohibitive latency in long video generation. A cross-modal audio-visual memory bank preserves character appearance and voice timbre consistently over videos, while a post-training pipeline combining memory-based reinforcement learning with distribution matching distillation (DMD) delivers a without sacrificing quality. JoyAI-Echo decisively outperforms (directing mode) on long-form generation and even surpasses the short-video specialist on human-centric tasks. This repository hosts the . Inference code is released separately — see the section. ## Model Details - Echo Team @ Joy Future Academy, JD - Text-to-(Audio+Video) diffusion transformer, DMD 8-step - Text → synchronized video + audio - Built on top of - (downloaded separately) - 1280 × 736, 241 frames @ 25 fps per shot - up to 5 minutes (multi-shot) - LTX-2 Community License Agreement ## Highlights - 🎞️ : generate a sequence of coherent shots from one prompt JSON. - ⚡ : ~7.5× faster than the original pipeline. - 🔊 : one pipeline produces synchronized video and audio. - 🧠 : conditions each new shot on prior visual identity and voice context for story-level consistency. ## Demo Gallery Explore long-form and short-form JoyAI-Echo cases on the . 🍿 ## Usage Inference is run with the standalone inference repository. ### 1. Download the checkpoint Also download the Gemma text encoder: Expected layout: ### 2. Get the inference code Environment: (see the inference repo's / ). ### 3. Write a story prompt We provide prompt enhancers — system prompts that expand a short story or idea into well-formed shot prompts: for long, multi-shot video, and for single-shot short video. We running your input through the matching enhancer before inference; un-enhanced prompts tend to produce noticeably weaker results. Create a JSON file under . Each file is a single object with a list, where . A single string produces one shot; multiple strings produce a multi-shot story, with each new shot conditioned on the previous ones through the paired audio-video memory bank. Inside each string, write these parts in order: | Part | What to describe | | --- | --- | | | Describe the appearance of all visible people, including age, build, hair, face, wardrobe, and speaking voice timbre when applicable. | | | What the subject does and speaks. | | | The overall visual and emotional aesthetic — e.g. realistic motorsport film language, cool daylight, restrained cinematic tension. | | | The shot type and framing or movement — e.g. a stable close-up on the face, or a medium shot from the waist up. | | | The setting and scene details behind the subject. | | | The sounds in the scene and the background music — e.g. room tone, wind, footsteps and fabric, with a soft low music bed under the dialogue or no background music. | ### 4. Run Outputs land in . ## Hardware Peak GPU memory is at the default 1280 × 736 × 241 frame setting — a single H100/A100 (80 GB) or 48 GB GPU is sufficient. For smaller GPUs, lower resolution or frame count: ## Results ### Reported Scale | Item | Value | | --- | ---: | | 🎬 Long-form coherent story length | | | ⚡ Generation speedup over the original multi-step pipeline | | | 📚 Benchmark stories | | | 🎞️ Generated evaluation shots | | | 🕒 Frames per shot | | ### Human Evaluation GSB user study on long- and short-video generation. The numbers denote the percentage of user preferences. | Aspect (Long Video) | JoyAI-Echo | Tie | HappyOyster (Directing) | | --- | ---: | ---: | ---: | | Visual aesthetics | | 8.8% | 27.6% | | Audio quality | | 6.5% | 11.8% | | Prompt following | | 13.5% | 5.9% | | IP consistency | | 12.9% | 27.7% | | Aspect (Short Video) | JoyAI-Echo | Tie | Wan 2.6 | | --- | ---: | ---: | ---: | | Visual aesthetics | | 14.7% | 26.5% | | Audio quality | 32.3% | 30.9% | 36.8% | | Prompt following | 33.8% | 36.8% | 29.4% | ## Links - Project page: - Inference code: - HuggingFace: ## Acknowledgements We gratefully acknowledge the open-source projects this work builds upon — in particular for the base video generator and for the text encoder. Thanks to the broader research community whose contributions made this release possible. ## Citation If JoyAI-Echo helps your research or products, please cite: ## License This project is based on LTX-2 by Lightricks Ltd. Portions of the original LTX-2 codebase have been modified by JD.com for academic and research purposes only. This project is not intended for commercial use. For commercial use of LTX-2 or its derivatives, please contact Lightricks Ltd. All original copyright, license, patent, trademark, and attribution notices from LTX-2 are retained. This project remains subject to the LTX-2 Community License Agreement.