PAC VLA

RPT-VLA

This repository contains a Hugging Face-style export of RPT-VLA, a diffusion-action vision-language-action model for SimplerEnv evaluation. The model takes a language instruction and an RGB observation as input, then predicts 7-DoF robot actions of the form (x, y, z, roll, pitch, yaw, gripper).

The checkpoint is packaged for offline local loading: model weights, tokenizer files, configuration files, and dataset statistics are stored in a single top-level directory.

Model Summary

• Model type: Vision-language-action policy with a diffusion action decoder
• Base VLM: prism-dinosiglip-224px+7b
• Vision backbone: DINOv2 + SigLIP fused vision backbone
• Language model: Llama-2 7B backbone
• Action decoder: DiT-B diffusion action head
• Action dimension: 7
• Future action window size: 15
• Past action window size: 6
• Repeated diffusion steps: 4
• Supported normalization keys: fractal20220817_data, bridge_dataset
• Checkpoint format: flat Hugging Face-style safetensors shards

Files

• config.json: model configuration used by the local diffusion-action loader
• config.yaml: original run configuration in YAML form
• dataset_statistics.json: action normalization statistics for supported evaluation datasets
• model.safetensors.index.json: index mapping parameters to safetensors shards
• model-*.safetensors: sharded model weights
• tokenizer.model, tokenizer.json, tokenizer_config.json, special_tokens_map.json, added_tokens.json: tokenizer files
• llama2_7b_config.json: local Llama-2 backbone configuration used for offline model construction
• offline_backbones.py: offline construction helper for local evaluation environments
• model_format.json: lightweight metadata describing the exported checkpoint layout

Uses

This model is intended for SimplerEnv robot policy evaluation and related fine-tuning workflows. It can be evaluated on task families whose action normalization statistics are included in dataset_statistics.json.

For execution, actions must be un-normalized with the correct unnorm_key:

• fractal20220817_data for Google Robot-style tasks
• bridge_dataset for WidowX / Bridge-style tasks

Using the wrong normalization key can produce invalid actions and unreliable evaluation results.

Getting Started

Use the accompanying project evaluation scripts and pass this directory as --model-dir.

Example for Google Robot-style tasks:

python scripts/eval.py \
  --robot google_robot \
  --model-dir /path/to/model \
  --simpler-env-root /path/to/simpler_env \
  --log-root /path/to/output/google_robot \
  --gpu 0 \
  --families ALL \
  --protocol both

Example for WidowX / Bridge-style tasks:

python scripts/eval.py \
  --robot widowx_robot \
  --model-dir /path/to/model \
  --simpler-env-root /path/to/simpler_env \
  --log-root /path/to/output/widowx_robot \
  --gpu 0

Evaluation Arguments

The evaluation entry is scripts/eval.py. The main configurable arguments are:

Argument	Default	Choices / Type	Description
--model-dir	../model	path	Path to this HF-style RPT-VLA model directory or another compatible model directory.
--simpler-env-root	SIMPLERENV_ROOT or ./SimplerEnv	path	Path to the local SimplerEnv checkout.
--log-root	../logs/eval	path	Output directory for progress.jsonl, progress_summary.txt, and videos/.
--gpu	0	string / int	GPU id passed as CUDA_VISIBLE_DEVICES to the evaluation process.
--robot	google_robot	google_robot, widowx_robot	Selects the evaluation benchmark.
--families	pick_coke_can,move_near,drawer	comma-separated list or ALL	Google Robot task families to evaluate. Used only when --robot google_robot.
--protocol	both	variant, visual_matching, both	Google Robot evaluation protocol subset. Used only when --robot google_robot.
--dry-run	disabled	flag	Print the commands that would run without starting evaluation.
--stop-on-error	disabled	flag	Stop after the first failed task subprocess.
--append-progress	disabled	flag	Append to an existing output directory instead of clearing previous progress files.
--save-videos	true	true / false	Save rollout videos.
--save-action-images	true	true / false	Save action visualization images.
--print-step-info	false	true / false	Print per-step simulator info to stdout.
--use-bf16	true	true / false	Use bfloat16 for the VLM backbone.
--cfg-scale	1.5	float	Classifier-free guidance scale for diffusion action sampling.
--num-ddim-steps	10	int	Number of DDIM sampling steps.
--use-ddim	true	true / false	Use DDIM sampling for the diffusion action decoder.
--action-ensemble	true	true / false	Enable temporal action ensembling.
--action-ensemble-horizon	robot default	int or unset	Override the default ensemble horizon. Defaults to 2 for Google Robot and 7 for WidowX.
--adaptive-ensemble-alpha	0.1	float	Weighting factor used by adaptive action ensembling.
--future-action-window-size	model config	int or unset	Override the model future action window size. Leave unset for normal evaluation.
--past-action-window-size	model config	int or unset	Override the model past action window size. Leave unset for normal evaluation.
--unnorm-key	robot default	string or unset	Override action normalization statistics. Defaults to fractal20220817_data for Google Robot and bridge_dataset for WidowX.

Google Robot supports these task families:

• pick_coke_can
• move_near
• drawer
• put_in_drawer

WidowX evaluation currently runs the bundled Bridge task set and does not use --families or --protocol.

Each evaluation output directory contains:

• progress.jsonl: rollout-level results
• progress_summary.txt: aggregate success rates
• videos/: rollout videos and action visualizations, if enabled

Limitations

The model is designed for the robot embodiments and evaluation setups represented by its normalization statistics and training data. It should not be assumed to transfer directly to unseen robot embodiments or substantially different camera/action conventions without additional validation or fine-tuning.