OmniNFT

--- license: apache-2.0 base_model: - Lightricks/LTX-2 - Lightricks/LTX-2.3 pipeline_tag: any-to-any library_name: peft tags: - lora - reinforcement-learning - GRPO - T2AV --- # 🎬 OmniNFT RL-LoRA for LTX Video > 🚀 Modality-wise Omni Diffusion Negative-aware Fine-Tuning for **LTX-2** and **LTX-2.3**~ <a href="https://zghhui.github.io/OmniNFT/"><img src="https://img.shields.io/badge/🌐-Project%20Page-blue" alt="Project Page"/></a> [](https://github.com/zghhui/OmniNFT) [](https://opensource.org/licenses/Apache-2.0) --- ## 📁 Model Files | Folder | Base Model | Description | |--------|-----------|-------------| | `LTX-2-RL-Lora/` | [Lightricks/LTX-2](https://huggingface.co/Lightricks/LTX-2) | RL-LoRA for LTX-2 | | `LTX-2.3-RL-Lora/` | [Lightricks/LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) | RL-LoRA for LTX-2.3 | --- ## 🛠️ Quick Start ### 1️⃣ Clone the Code ```bash git clone https://github.com/zghhui/OmniNFT.git cd OmniNFT pip install -r requirements.txt ``` ### 2️⃣ Download Base Models Download the base checkpoints from HuggingFace: ```bash # LTX-2 huggingface-cli download Lightricks/LTX-2 --local-dir ./checkpoints/LTX-2 # LTX-2.3 huggingface-cli download Lightricks/LTX-2.3 --local-dir ./checkpoints/LTX-2.3 ``` ### 3️⃣ Download RL-LoRA Weights ```bash # LTX-2 LoRA huggingface-cli download zghhui/OmniNFT \ --include "LTX-2-RL-Lora/*" \ --local-dir ./checkpoints/ # LTX-2.3 LoRA huggingface-cli download zghhui/OmniNFT \ --include "LTX-2.3-RL-Lora/*" \ --local-dir ./checkpoints/ ``` ### 4️⃣ Merge LoRA into Base Model ```bash # 🔀 Merge LTX-2 + RL-LoRA python scripts/merge_lora.py \ --checkpoint-path ./checkpoints/LTX-2/transformer/model.safetensors \ --lora-dir ./checkpoints/LTX-2-RL-Lora \ --output-path ./merged_model_ltx2.safetensors \ --dtype bf16 # 🔀 Merge LTX-2.3 + RL-LoRA python scripts/merge_lora.py \ --checkpoint-path ./checkpoints/LTX-2.3/transformer/model.safetensors \ --lora-dir ./checkpoints/LTX-2.3-RL-Lora \ --output-path ./merged_model_ltx23.safetensors \ --dtype bf16 ``` ### 5️⃣ Inference 🎥 ```bash python scripts/inference.py \ --model_path ./merged_model_ltx2.safetensors \ --gemma_path $GEMMA_MODEL_PATH \ --prompt "A man plays acoustic guitar on a wooden sta...