Act Banana In Pot

ACT — Put the right banana in the pot (UR7e)

An Action Chunking Transformer (ACT) policy trained by imitation learning to perform the manipulation task "put the right banana in the pot" on a Universal Robots UR7e arm with two RGB cameras.

• Policy: ACT (Transformer encoder–decoder with a CVAE), ResNet18 vision backbone (ImageNet-pretrained), chunk_size = 100, n_action_steps = 100.
• Trained on: Bigenlight/banana_in_pot_lerobot_v3 (51 teleoperated episodes / 21,524 frames, UR7e follower + GELLO leader, 2 cameras).
• Training length: 50,000 steps, batch size 8, AdamW lr = 1e-5 (constant).
• Framework: LeRobot v0.6.1.

What the policy does

I/O	Spec
observation.state	(7,) — UR joints q1..q6 (radians) + gripper position
observation.images.cam1 / cam2	RGB, trained at 360×640
action	(7,) — [cmd1..cmd6, grip_cmd], absolute joint targets (radians) + ~binary gripper

Images were captured at 720p and resized on-the-fly to 360×640 during training (aspect-preserving half-resolution; no dataset re-encode). You must resize every live camera frame to 360×640 at inference — a different aspect ratio or interpolation degrades the policy.

Data & hardware setup

Component	Detail
Robot	Universal Robots UR7e — 6-DOF collaborative arm, joints in radians. Inference uses the UR7e follower only.
Teleoperation (data collection)	GELLO low-cost 3D-printed leader arm. Leader signals are recorded but are not policy inputs at inference.
Camera 1	Intel RealSense D435 — RGB only
Camera 2	Intel RealSense D435if — RGB only
Camera streams	1280×720 (720p) @ 30 fps, color only (no depth / IR recorded). ACT is trained on the two RGB views resized to 360×640.
Task	"put the right banana in the pot" — table with distractor objects (2 bananas, apple, carrots/peppers, watermelon slice) and a silver pot; success = the correct banana placed in the pot.
Dataset scale	51 episodes / 21,524 frames / ~12 min @ 30 fps.

Training configuration

Item	Value
Policy	ACT (ResNet18 backbone, CVAE, chunk_size=100, n_action_steps=100)
dim_model / heads	512 / 8
Encoder / decoder layers	4 / 1
Normalization	MEAN_STD for state, action, and visual
Batch / steps	8 / 50,000
Optimizer	AdamW, lr = 1e-5 (constant), backbone lr = 1e-5
Backbone	ResNet18_Weights.IMAGENET1K_V1 (ImageNet-pretrained)
GPU	RTX 3060 12GB (~4.7 GB used, ~3.9 step/s)

Training loss

training loss

Final training loss ≈ 0.065.

Offline evaluation (open-loop: predicted actions vs. ground truth)

Evaluated on held-out episodes (train vs. held-out gap is negligible → generalizes, no overfitting). The step 50,000 checkpoint (this model) is the best by held-out L1.

step	joints MAE (rad)	overall L1	gripper acc	train L1	held-out L1
10000	0.0392	0.0395	98.0%	0.0392	0.0406
20000	0.0354	0.0345	98.8%	0.0324	0.0356
30000	0.0267	0.0265	98.8%	0.0258	0.0269
40000	0.0256	0.0245	99.2%	0.0222	0.0256
50000 ⭐	0.0237	0.0225	99.2%	0.0205	0.0235

evaluation trend

Best checkpoint (step 50,000):

• Held-out overall L1 = 0.0235 rad (≈ 1.34°); train L1 = 0.0205 → near-zero gap.
• Joints MAE = 0.0237 rad (≈ 1.36°).
• Gripper accuracy 99.2%.

per-joint MAE

The wrist joint (cmd6) carries the largest error — it is the axis with the most natural variation. The gripper channel is close to binary.

How to use

Load the policy (LeRobot)

from lerobot.policies.act import ACTPolicy
from lerobot.policies import make_pre_post_processors

policy = ACTPolicy.from_pretrained("Bigenlight/act_banana_in_pot")
policy.eval()

# Normalization is NOT baked into forward() in lerobot 0.6.1 — it lives in the
# processor pipeline saved alongside the checkpoint. select_action returns a
# NORMALIZED action; the post-processor converts it back to radians.
preprocessor, postprocessor = make_pre_post_processors(
    policy_cfg=policy.config,
    pretrained_path="Bigenlight/act_banana_in_pot",
)

Per control tick:

policy.reset()                       # once at the start of each rollout
obs = preprocessor(obs)              # normalize + batch + move to device
action = policy.select_action(obs)   # normalized
action = postprocessor(action)       # radians, numpy (7,)

select_action returns one action per call from an internal queue; on an empty queue it predicts a full 100-step chunk and replans after 100 executed actions (temporal ensembling is off by default in this config).

Deploy on a real UR7e

LeRobot ships no UR robot class — you build the observation dict yourself and stream joint targets with ur_rtde at 30 Hz (33.3 ms control period). Loop: read getActualQ() + gripper → grab both cameras, BGR→RGB, resize to 360×640 → build observation.state / observation.images.cam1 / cam2 → preprocess → select_action → postprocess → servoJ(q_target[:6]) + drive gripper from grip_cmd.

⚠️ Safety — actions are ABSOLUTE joint positions (the single biggest safety driver):

1. Start near the dataset initial pose q1..q6 ≈ [2.84, -1.41, 1.78, -2.01, -1.66, -3.42] rad before enabling the policy, or the first absolute command is a large jump.
2. First-command jump guard: if max(|q_target − getActualQ()|) > ~0.15 rad, abort.
3. Clamp per-tick joint change (e.g. ≤ 0.05–0.1 rad / 33 ms early on) and clamp to UR software joint limits.
4. Reduced speed for first trials; keep a hand on the E-stop dead-man switch.
5. Camera mapping matters: the policy learned a fixed cam1/cam2 → physical-viewpoint mapping. Swap them and it fails silently. Verify wiring every session.
6. Gripper grip_cmd is ~binary — threshold (e.g. >0.5 → close) and map to your driver.

A full deployment guide (with a reference ur_rtde skeleton and all the citations to the lerobot inference/normalization code paths) is in the project's DEPLOY_UR.md.

Related repositories

• Dataset: Bigenlight/banana_in_pot_lerobot_v3
• HIL-SERL prep bundle (reward classifier, SAC config, runbook) for taking this task online with reinforcement learning: Bigenlight/banana_in_pot_hilserl

Limitations

• Small dataset (51 demos). Metrics reflect an easy, temporally-correlated task; expect reduced robustness to novel object positions, lighting, or camera placement.
• Offline metrics only. All numbers above are open-loop (predicted vs. ground-truth actions). Real closed-loop task success on the arm has not been measured here — a physical UR7e deployment is required for the final verdict.
• Absolute-joint action space demands the safety guards above; the policy was only ever conditioned on states near the data-collection start pose.
• The ResNet18 backbone is ImageNet-pretrained (not robotics-pretrained); the CVAE and transformer are trained from scratch on this task.