--- library_name: diffusion-action-vla tags: - robotics - vla - diffusion-policy - multimodal - image-text-to-action language: - en pipeline_tag: robotics --- # RPT-VLA This repository contains a Hugging Face-style export of RPT-VLA, a diffusion-action vision-language-action model for SimplerEnv evaluation. The model takes a language instruction and an RGB observation as input, then predicts 7-DoF robot actions of the form `(x, y, z, roll, pitch, yaw, gripper)`. The checkpoint is packaged for offline local loading: model weights, tokenizer files, configuration files, and dataset statistics are stored in a single top-level directory. ## Model Summary - **Model type:** Vision-language-action policy with a diffusion action decoder - **Base VLM:** `prism-dinosiglip-224px+7b` - **Vision backbone:** DINOv2 + SigLIP fused vision backbone - **Language model:** Llama-2 7B backbone - **Action decoder:** DiT-B diffusion action head - **Action dimension:** 7 - **Future action window size:** 15 - **Past action window size:** 6 - **Repeated diffusion steps:** 4 - **Supported normalization keys:** `fractal20220817_data`, `bridge_dataset` - **Checkpoint format:** flat Hugging Face-style `safetensors` shards ## Files - `config.json`: model configuration used by the local diffusion-action loader - `config.yaml`: original run configuration in YAML form - `dataset_statistics.json`: action normalization statistics for supported evaluation datasets - `model.safetensors.index.json`: index mapping parameters to safetensors shards - `model-*.safetensors`: sharded model weights - `tokenizer.model`, `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, `added_tokens.json`: tokenizer files - `llama2_7b_config.json`: local Llama-2 backbone configuration used for offline model construction - `offline_backbones.py`: offline construction helper for local evaluation environments - `model_format.json`: lightweight metadata describing the exported checkpoint layout ## Uses This model is intended for SimplerEnv robot policy evaluation and related fine-tuning workflows. It can be evaluated on task families whose action normalization statistics are included in `dataset_statistics.json`. For execution, actions must be un-normalized with the correct `unnorm_key`: - `fractal20220817_data` for Google Robot-style tasks - `bridge_dataset` for WidowX / Bridge-style tasks Using the wrong normalization key can produce invalid actions and unreliable evaluation results. ## Getting Started Use the accompanying project evaluation scripts and pass this directory as `--model-dir`. Example for Google Robot-style tasks: ```bash python scripts/eval.py \ --robot google_robot \ --model-dir /path/to/model \ --simpler-env-root /path/to/simpler_env \ --log-root /path/to/output/google_robot \ --gpu 0 \ --families ALL \ --protocol both ``` Example for WidowX / Bridge-style tasks: ```bash python scripts/eval.py \ --robot widowx_robot \ --model-dir /path/to/model \ --simpler-env-root /path/to/simpler_env \ --log-root /path/to/output/widowx_robot \ --gpu 0 ``` ## Evaluation Arguments The evaluation entry is `scripts/eval.py`. The main configurable arguments are: | Argument | Default | Choices / Type | Description | | --- | --- | --- | --- | | `--model-dir` | `../model` | path | Path to this HF-style RPT-VLA model directory or another compatible model directory. | | `--simpler-env-root` | `SIMPLERENV_ROOT` or `./SimplerEnv` | path | Path to the local SimplerEnv checkout. | | `--log-root` | `../logs/eval` | path | Output directory for `progress.jsonl`, `progress_summary.txt`, and `videos/`. | | `--gpu` | `0` | string / int | GPU id passed as `CUDA_VISIBLE_DEVICES` to the evaluation process. | | `--robot` | `google_robot` | `google_robot`, `widowx_robot` | Selects the evaluation benchmark. | | `--families` | `pick_coke_can,move_near,drawer` | comma-separated list or `ALL` | Google Robot task families to evaluate. Used only when `--robot google_robot`. | | `--protocol` | `both` | `variant`, `visual_matching`, `both` | Google Robot evaluation protocol subset. Used only when `--robot google_robot`. | | `--dry-run` | disabled | flag | Print the commands that would run without starting evaluation. | | `--stop-on-error` | disabled | flag | Stop after the first failed task subprocess. | | `--append-progress` | disabled | flag | Append to an existing output directory instead of clearing previous progress files. | | `--save-videos` | `true` | `true` / `false` | Save rollout videos. | | `--save-action-images` | `true` | `true` / `false` | Save action visualization images. | | `--print-step-info` | `false` | `true` / `false` | Print per-step simulator info to stdout. | | `--use-bf16` | `true` | `true` / `false` | Use bfloat16 for the VLM backbone. | | `--cfg-scale` | `1.5` | float | Classifier-free guidance scale for diffusion action sampling. | | `--num-ddim-steps` | `10` | int | Number of DDIM sampling steps. | | `--use-ddim` | `true` | `true` / `false` | Use DDIM sampling for the diffusion action decoder. | | `--action-ensemble` | `true` | `true` / `false` | Enable temporal action ensembling. | | `--action-ensemble-horizon` | robot default | int or unset | Override the default ensemble horizon. Defaults to `2` for Google Robot and `7` for WidowX. | | `--adaptive-ensemble-alpha` | `0.1` | float | Weighting factor used by adaptive action ensembling. | | `--future-action-window-size` | model config | int or unset | Override the model future action window size. Leave unset for normal evaluation. | | `--past-action-window-size` | model config | int or unset | Override the model past action window size. Leave unset for normal evaluation. | | `--unnorm-key` | robot default | string or unset | Override action normalization statistics. Defaults to `fractal20220817_data` for Google Robot and `bridge_dataset` for WidowX. | Google Robot supports these task families: - `pick_coke_can` - `move_near` - `drawer` - `put_in_drawer` WidowX evaluation currently runs the bundled Bridge task set and does not use `--families` or `--protocol`. Each evaluation output directory contains: - `progress.jsonl`: rollout-level results - `progress_summary.txt`: aggregate success rates - `videos/`: rollout videos and action visualizations, if enabled ## Limitations The model is designed for the robot embodiments and evaluation setups represented by its normalization statistics and training data. It should not be assumed to transfer directly to unseen robot embodiments or substantially different camera/action conventions without additional validation or fine-tuning.
