--- license: apache-2.0 library_name: lerobot tags: - act - robotics - lerobot - ur7e - imitation-learning pipeline_tag: robotics datasets: - Bigenlight/banana_in_pot_lerobot_v3 --- # ACT — Put the right banana in the pot (UR7e) An **Action Chunking Transformer (ACT)** policy trained by imitation learning to perform the manipulation task *"put the right banana in the pot"* on a **Universal Robots UR7e** arm with two RGB cameras. - **Policy:** ACT (Transformer encoder–decoder with a CVAE), **ResNet18** vision backbone (ImageNet-pretrained), `chunk_size = 100`, `n_action_steps = 100`. - **Trained on:** [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3) (51 teleoperated episodes / 21,524 frames, UR7e follower + GELLO leader, 2 cameras). - **Training length:** 50,000 steps, batch size 8, AdamW `lr = 1e-5` (constant). - **Framework:** [LeRobot](https://github.com/huggingface/lerobot) v0.6.1. ## What the policy does | I/O | Spec | |---|---| | `observation.state` | `(7,)` — UR joints `q1..q6` (radians) + gripper position | | `observation.images.cam1` / `cam2` | RGB, trained at **360×640** | | `action` | `(7,)` — `[cmd1..cmd6, grip_cmd]`, **absolute** joint targets (radians) + ~binary gripper | Images were captured at 720p and **resized on-the-fly to 360×640** during training (aspect-preserving half-resolution; no dataset re-encode). **You must resize every live camera frame to 360×640** at inference — a different aspect ratio or interpolation degrades the policy. ## Data & hardware setup | Component | Detail | |---|---| | Robot | **Universal Robots UR7e** — 6-DOF collaborative arm, joints in radians. Inference uses the UR7e follower only. | | Teleoperation (data collection) | **GELLO** low-cost 3D-printed leader arm. Leader signals are recorded but are **not** policy inputs at inference. | | Camera 1 | **Intel RealSense D435** — RGB only | | Camera 2 | **Intel RealSense D435if** — RGB only | | Camera streams | 1280×720 (720p) @ **30 fps**, color only (**no depth / IR** recorded). ACT is trained on the two RGB views **resized to 360×640**. | | Task | *"put the right banana in the pot"* — table with distractor objects (2 bananas, apple, carrots/peppers, watermelon slice) and a silver pot; success = the correct banana placed in the pot. | | Dataset scale | 51 episodes / 21,524 frames / ~12 min @ 30 fps. | ## Training configuration | Item | Value | |---|---| | Policy | ACT (ResNet18 backbone, CVAE, `chunk_size=100`, `n_action_steps=100`) | | `dim_model` / heads | 512 / 8 | | Encoder / decoder layers | 4 / 1 | | Normalization | MEAN_STD for state, action, and visual | | Batch / steps | 8 / 50,000 | | Optimizer | AdamW, `lr = 1e-5` (constant), backbone `lr = 1e-5` | | Backbone | `ResNet18_Weights.IMAGENET1K_V1` (ImageNet-pretrained) | | GPU | RTX 3060 12GB (~4.7 GB used, ~3.9 step/s) | ### Training loss  Final training loss ≈ **0.065**. ## Offline evaluation (open-loop: predicted actions vs. ground truth) Evaluated on held-out episodes (train vs. held-out gap is negligible → generalizes, no overfitting). The **step 50,000** checkpoint (this model) is the best by held-out L1. | step | joints MAE (rad) | overall L1 | gripper acc | train L1 | held-out L1 | |---|---|---|---|---|---| | 10000 | 0.0392 | 0.0395 | 98.0% | 0.0392 | 0.0406 | | 20000 | 0.0354 | 0.0345 | 98.8% | 0.0324 | 0.0356 | | 30000 | 0.0267 | 0.0265 | 98.8% | 0.0258 | 0.0269 | | 40000 | 0.0256 | 0.0245 | 99.2% | 0.0222 | 0.0256 | | **50000** ⭐ | **0.0237** | **0.0225** | **99.2%** | 0.0205 | **0.0235** |  **Best checkpoint (step 50,000):** - Held-out overall **L1 = 0.0235 rad** (≈ 1.34°); train L1 = 0.0205 → near-zero gap. - Joints **MAE = 0.0237 rad** (≈ 1.36°). - Gripper accuracy **99.2%**.  The wrist joint (`cmd6`) carries the largest error — it is the axis with the most natural variation. The gripper channel is close to binary. ## How to use ### Load the policy (LeRobot) ```python from lerobot.policies.act import ACTPolicy from lerobot.policies import make_pre_post_processors policy = ACTPolicy.from_pretrained("Bigenlight/act_banana_in_pot") policy.eval() # Normalization is NOT baked into forward() in lerobot 0.6.1 — it lives in the # processor pipeline saved alongside the checkpoint. select_action returns a # NORMALIZED action; the post-processor converts it back to radians. preprocessor, postprocessor = make_pre_post_processors( policy_cfg=policy.config, pretrained_path="Bigenlight/act_banana_in_pot", ) ``` Per control tick: ```python policy.reset() # once at the start of each rollout obs = preprocessor(obs) # normalize + batch + move to device action = policy.select_action(obs) # normalized action = postprocessor(action) # radians, numpy (7,) ``` `select_action` returns one action per call from an internal queue; on an empty queue it predicts a full 100-step chunk and replans after 100 executed actions (temporal ensembling is **off** by default in this config). ### Deploy on a real UR7e LeRobot ships **no UR robot class** — you build the observation dict yourself and stream joint targets with [`ur_rtde`](https://sdurobotics.gitlab.io/ur_rtde/) at **30 Hz** (33.3 ms control period). Loop: read `getActualQ()` + gripper → grab both cameras, BGR→RGB, resize to 360×640 → build `observation.state` / `observation.images.cam1` / `cam2` → preprocess → `select_action` → postprocess → `servoJ(q_target[:6])` + drive gripper from `grip_cmd`. **⚠️ Safety — actions are ABSOLUTE joint positions (the single biggest safety driver):** 1. **Start near the dataset initial pose** `q1..q6 ≈ [2.84, -1.41, 1.78, -2.01, -1.66, -3.42]` rad before enabling the policy, or the first absolute command is a large jump. 2. **First-command jump guard:** if `max(|q_target − getActualQ()|) > ~0.15 rad`, **abort**. 3. **Clamp per-tick joint change** (e.g. ≤ 0.05–0.1 rad / 33 ms early on) and clamp to UR software joint limits. 4. **Reduced speed for first trials**; keep a hand on the **E-stop** dead-man switch. 5. **Camera mapping matters:** the policy learned a fixed `cam1`/`cam2` → physical-viewpoint mapping. Swap them and it fails silently. Verify wiring every session. 6. **Gripper** `grip_cmd` is ~binary — threshold (e.g. `>0.5 → close`) and map to your driver. A full deployment guide (with a reference `ur_rtde` skeleton and all the citations to the lerobot inference/normalization code paths) is in the project's `DEPLOY_UR.md`. ## Related repositories - **Dataset:** [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3) - **HIL-SERL prep bundle** (reward classifier, SAC config, runbook) for taking this task online with reinforcement learning: [`Bigenlight/banana_in_pot_hilserl`](https://huggingface.co/Bigenlight/banana_in_pot_hilserl) ## Limitations - **Small dataset** (51 demos). Metrics reflect an easy, temporally-correlated task; expect reduced robustness to novel object positions, lighting, or camera placement. - **Offline metrics only.** All numbers above are open-loop (predicted vs. ground-truth actions). Real closed-loop task success on the arm has not been measured here — a physical UR7e deployment is required for the final verdict. - **Absolute-joint action space** demands the safety guards above; the policy was only ever conditioned on states near the data-collection start pose. - The ResNet18 backbone is **ImageNet-pretrained** (not robotics-pretrained); the CVAE and transformer are trained from scratch on this task.
