📖 Documentation | 🤗 Model Hub ## 1. Model Introduction FluxVLA Engine is an integrated engineering platform designed for embodied intelligence applications. It follows the core design principles of unified configuration, standardized interfaces, module decoupling, and deployability, forming a complete engineering loop from data collection to real-world deployment. With a focus on building a "standardized industrial-academic-research foundation," FluxVLA significantly lowers the engineering threshold for VLA (Visual Language Agent) research and development. ### Key Features 🧩 **Rich VLA Model Zoo**: Built-in support for mainstream VLA families including OpenVLA, LLaVA-VLA, GR00T, Pi0, Pi0.5, and DreamZero, enabling fast comparison, fine-tuning, and deployment across different model architectures. ⚡ **Extreme Inference Speed**: CUDA-accelerated operator fusion. With the GR00T-N1.5, the system achieves a high-frequency of 42.8Hz on the RTX 5090. 🛠 **All-in-One**: A unified configuration file manages critical parameters for data, models, training, evaluation, inference, and deployment, ensuring easy reproducibility and deployment. 🖥 **Flexible Configuration**: Supports mainstream visual backbones, LLM (Large Language Models), VLM (Vision-Language Models), and VLA (Vision-Language Agents). You can also create and customize your own VLA. 🤖 **Real-world Deployment Ready**: In addition to mainstream simulators, FluxVLA comes with ready-to-use deployment scripts for Aloha, Tron2, and UR3, eliminating the need for custom integration. 💨 **Smooth Motion**: Optimized for real-world training/inference with RTC trajectory guidance, ensuring accurate and continuous motion even at 40Hz+ frequencies. 📊 **Strong Benchmark Performance**: Provides competitive LIBERO benchmark results across spatial, object, goal, and long-horizon tasks, making it easier to evaluate VLA models under standardized settings. 🛰 **Remote Inference Ready**: Includes a ZMQ-based server/client inference framework, enabling GPU-offloaded remote inference for resource-constrained robot-side devices. 🧪 **Scalable Training Pipeline**: Supports distributed training with DDP/FSDP, LoRA fine-tuning, eval-after-train, and checkpoint-based resume, covering both local debugging and cluster-scale training workflows. ## 2. Model Zoo FluxVLA Engine supports multiple VLA model families, each optimized for different use cases: | Family | Parameters | Key Features | Typical Use Cases | |:---:|:---:|:---|:---| | **GR00T** | 3B | High-frequency action prediction, CUDA-optimized | Real-time robot control, deployment | | **Pi0 / Pi0.5** | 3B | Flow-matching VLA, smooth trajectories | Continuous control, manipulation tasks | | **DreamZero** | 23B | Diffusion/flow-based generation | Complex motion planning, LIBERO tasks | | **LLaVA-VLA** | 3B-7B | VLM-based architecture | Vision-language grounding, instruction following | | **OpenVLA** | 7B | Open-source baseline | Benchmark comparison, fine-tuning | ### Available Pretrained Models | Model | Size | Download Link | |:---:|:---:|:---| | GR00T N1.5 | 3B | [Hugging Face](https://huggingface.co/nvidia/GR00T-N1.5-3B) | | PI0 Base | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi0_base) | | PI0.5 Base | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_base) | | PI0.5 LIBERO | 3B | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_libero) | | OpenVLA | 7B | [Hugging Face](https://huggingface.co/openvla/openvla-7b-finetuned-libero-10) | ### Trained FluxVLA Checkpoints | Model | Training Data | Performance | Download | |:---:|:---:|:---:|:---| | PI0.5 PaliGemma | LIBERO-10 | 96.0% avg | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/pi05_paligemma_libero_10_full_finetune_bs64) | | GR00T Eagle 3B | LIBERO-10 | 89.4% avg | [Hugging Face](https://huggingface.co/limxdynamics/FluxVLAEngine/tree/main/gr00t_eagle_3b_libero_10_full_finetune_bs64) | ## 3. Evaluation Results FluxVLA demonstrates state-of-the-art performance across multiple LIBERO benchmark suites: ### LIBERO Benchmark Performance | Model | Libero-Spatial | Libero-Object | Libero-Goal | Libero-Long | Average | |:---:|:---:|:---:|:---:|:---:|:---:| | **FluxVLA (Pi0.5)** | 98.6 | 99.0 | 97.8 | 96.0±1.0 | 97.85 | | **FluxVLA (Qwen3VL+GR00T)** | 98.6 | 99.6 | 95.6 | 92.2±1.8 | 96.50 | | **FluxVLA (GR00T)** | 96.2 | 96.8 | 93.4 | 89.4±1.5 | 93.95 | | **FluxVLA (DreamZero)** | 96.8 | 97.4 | 90.8±1.5 | 93.6 | 94.65 | ### Inference Speed Comparison | Model | Hardware | Inference Speed | Acceleration | |:---:|:---:|:---:|:---:| | GR00T N1.5 (Optimized) | RTX 5090 | 42.8 Hz | 3.2x | | GR00T N1.5 (Baseline) | RTX 5090 | 13.4 Hz | 1.0x | | Pi0.5 (Optimized) | RTX 5090 | 38.5 Hz | 2.8x | | Pi0.5 (Baseline) | RTX 5090 | 13.7 Hz | 1.0x | *Note: Optimized versions use CUDA-accelerated operator fusion and Triton kernels.* ## 4. Architecture Overview FluxVLA Engine supports flexible architecture combinations: ### Supported Components | Component Type | Options | |:---:|:---| | **Vision Encoders** | DINOv2 ViT-Large, SigLIP ViT-SO400M, SigLIP2, PaliGemma | | **Language Models** | Qwen2.5 (3B/7B), Llama 2 (7B) | | **Vision-Language Models** | Qwen2.5-VL (3B), PaliGemma (3B) | | **Action Heads** | Diffusion, Flow Matching, Direct Regression | | **Precision** | FP32, FP16, BF16| ## 5. Installation & Deployment ### Quick Installation ```bash # Clone repository git clone https://github.com/limxdynamics/FluxVLA.git cd FluxVLA # Create environment conda create -n fluxvla python=3.10 -y conda activate fluxvla # Install PyTorch (CUDA 12.4) pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \ --index-url https://download.pytorch.org/whl/cu124 # Install flash-attention pip install psutil ninja packaging MAX_JOBS=8 pip install flash-attn==2.5.5 --no-build-isolation # Install FluxVLA pip install -r requirements.txt pip install --no-build-isolation -e . ``` ### Deployment Options FluxVLA supports multiple deployment scenarios: 1. **Local Training & Evaluation**: Single-node or multi-node distributed training with DDP/FSDP 2. **Real-Robot Deployment**: Direct on-robot inference with optimized runtime 3. **Remote Inference**: GPU server + robot client architecture via ZMQ 4. **Simulation**: LIBERO, Isaac Sim integration ### Hardware Requirements | Deployment Type | Minimum GPU | Recommended GPU | Memory | |:---:|:---:|:---:|:---:| | Training (3B model) | RTX 3090 (24GB) | A100 (40GB) | 32GB+ RAM | | Inference (optimized) | RTX 4090 (24GB) | RTX 5090 (32GB) | 16GB+ RAM | | Remote Inference Server | RTX 4090 (24GB) | RTX 5090 (32GB) | 16GB+ RAM | | Robot Client | CPU only | - | 8GB+ RAM | ## 6. Model Usage ### Training Train a VLA model on LIBERO-10 dataset: ```bash # Single-node training torchrun \ --standalone \ --nnodes 1 \ --nproc-per-node 2 \ scripts/train.py \ --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \ --work-dir ./checkpoints/pi05_paligemma_libero_10_full_finetune \ --cfg-options train_dataloader.per_device_batch_size=2 # Or use the launcher script bash scripts/train.sh \ configs/gr00t/gr00t_eagle_3b_libero_10_full_finetune.py \ work_dirs/gr00t_eagle_3b_libero_10_full_finetune ``` ### Evaluation Evaluate a trained model on LIBERO benchmarks: ```bash # Evaluate on LIBERO-10 torchrun \ --standalone \ --nnodes 1 \ --nproc-per-node 2 \ scripts/eval.py \ --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \ --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors # Or use the launcher script bash scripts/eval.sh \ configs/pi05/pi05_paligemma_libero_10_full_finetune.py \ checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors ``` ### Inference #### Simulation Inference ```python import torch from fluxvla.models import build_model from fluxvla.datasets import build_dataset from fluxvla.transforms import build_transforms # Load configuration config = load_config('configs/pi05/pi05_paligemma_libero_10_full_finetune.py') # Build model model = build_model(config.model) model.load_checkpoint('path/to/checkpoint.safetensors') model.eval() model.cuda() # Build transforms transforms = build_transforms(config.transforms) # Inference loop with torch.no_grad(): # Prepare input image = transforms['image'](raw_image) instruction = "Pick up the red cube" # Get action prediction action = model.predict( image=image, instruction=instruction, proprioception=robot_state ) # Execute action on robot robot.execute(action) ``` #### Real-Robot Inference ```bash # Direct on-robot inference python scripts/inference_real_robot.py \ --config configs/pi05/pi05_paligemma_aloha_full_finetune.py \ --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors ``` #### Remote Inference For deployment scenarios where the robot cannot host the full model: ```bash # On GPU server python scripts/remote_inference_server.py \ --config configs/pi05/pi05_paligemma_aloha_remote_inference.py \ --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/latest-checkpoint.safetensors \ --host 0.0.0.0 \ --port 5555 # On robot client bash scripts/remote_inference_client.sh ``` ## 7. Advanced Features ### RTC (Real-Time Control) Trajectory Guidance FluxVLA includes RTC trajectory guidance for smooth, continuous motion control: RTC improves action continuity by conditioning on recent action history, reducing jitter and improving success rates in real-world deployment. ```python # Enable RTC in config model = dict( type='Pi05VLA', rtc_enabled=True, rtc_prefix_length=5, rtc_temperature=0.1, ) ``` ### Inference Acceleration FluxVLA provides multiple acceleration techniques: 1. **CUDA Operator Fusion**: Custom CUDA kernels for attention and MLP layers 2. **Triton Kernels**: Optimized Triton implementations for key operations 3. **CUDA Graph**: Graph capture for reduced kernel launch overhead 4. **Mixed Precision**: FP16/BF16 inference with minimal accuracy loss ```bash # Enable acceleration in config python scripts/inference.py \ --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \ --ckpt-path checkpoints/latest.safetensors \ --use-cuda-graph \ --use-triton-kernels ``` ### Distributed Training FluxVLA supports both DDP and FSDP for multi-GPU training: ```bash # Multi-node training with FSDP torchrun \ --nnodes 4 \ --nproc-per-node 8 \ --rdzv-backend c10d \ --rdzv-endpoint $MASTER_ADDR:$MASTER_PORT \ scripts/train.py \ --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \ --work-dir ./checkpoints/distributed_training ``` ### LoRA Fine-tuning For efficient fine-tuning with limited compute: ```python # Enable LoRA in config lora = dict( enabled=True, r=16, lora_alpha=32, target_modules=['q_proj', 'v_proj', 'k_proj', 'o_proj'], lora_dropout=0.05, ) ``` ## 8. Supported Datasets FluxVLA supports multiple robot learning datasets: | Dataset | Tasks | Episodes | Download | |:---:|:---:|:---:|:---| | LIBERO-Spatial | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_spatial_no_noops_lerobotv2.1) | | LIBERO-Object | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_object_no_noops_lerobotv2.1) | | LIBERO-Goal | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_goal_no_noops_lerobotv2.1) | | LIBERO-10 | 10 | ~5K | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/libero_10_no_noops_lerobotv2.1) | | Aloha Real Robot | Custom | Variable | [Hugging Face](https://huggingface.co/datasets/limxdynamics/FluxVLAData/tree/main/RealRobot_AgileX_aloha_lerobot_v2) | | UR3 Real Robot | Custom | Variable | [Hugging...
