--- license: apache-2.0 library_name: lerobot tags: - hil-serl - reinforcement-learning - sac - rlpd - robotics - ur7e - lerobot pipeline_tag: robotics datasets: - Bigenlight/banana_in_pot_lerobot_v3 --- # HIL-SERL offline prep bundle — Put the right banana in the pot (UR7e) This repository is the **offline-prepared base for HIL-SERL** (Human-in-the-Loop Sample-Efficient Robotic reinforcement Learning) on the *"put the right banana in the pot"* task with a **Universal Robots UR7e**. It contains **everything up to online RL** — the vision reward classifier, the SAC/RLPD training config, the UR7e kinematics (URDF + FK), the offline demo buffer builder, and the runbook. The **online phase itself is robot-only** and is not included here (it requires the physical arm plus `ur_rtde` + `placo` + a gRPC actor/learner). Built from [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3) (51 teleop demos / 21,524 frames) using [LeRobot](https://github.com/huggingface/lerobot)'s HIL-SERL stack. ## Data & hardware setup | Component | Detail | |---|---| | Robot | **Universal Robots UR7e** — 6-DOF collaborative arm, joints in radians. | | Teleoperation (demos) | **GELLO** low-cost 3D-printed leader arm. | | Camera 1 | **Intel RealSense D435** — RGB only | | Camera 2 | **Intel RealSense D435if** — RGB only | | Camera streams | 1280×720 @ 30 fps, color only (**no depth / IR**). HIL-SERL uses the two RGB views **resized to 128×128**. | | Task | *"put the right banana in the pot"* — distractors + silver pot; success = correct banana in the pot. | ## What's in this bundle ``` banana_in_pot_hilserl/ ├── reward_classifier/ │ └── checkpoint/ # trained success classifier (config.json + model.safetensors) ├── config/ │ └── train_hilserl_ur7e.json # shared SAC/RLPD learner+actor config (paths relative) ├── build_offline_buffer.py # rebuilds the 54 MB RL demo buffer from the dataset ├── joint_to_ee.py # UR7e forward kinematics (Joint → EE), placo-free, validated ├── ur7e.urdf # pre-generated UR7e URDF (IK target frame `tool0`) ├── HILSERL_RUNBOOK.md # exact online start sequence (robot-only steps marked [ROBOT]) └── assets/ # figures embedded below ``` > **Offline demo buffer is NOT shipped** (it is ~54 MB of video/state transitions). Rebuild it > from the dataset with `build_offline_buffer.py` — see [Rebuilding the demo buffer](#rebuilding-the-offline-demo-buffer). ## 1. Reward / success classifier A vision-based binary success detector trained offline, in the exact LeRobot `reward_classifier` format so it drops straight into HIL-SERL. - **Encoder:** frozen `lerobot/resnet10` (CNN), per-camera `SpatialLearnedEmbeddings` pool + Linear→LayerNorm→Tanh. Following the official LeRobot behavior, the encoder runs under `no_grad` so **only the classifier head trains** (~2.36 M trainable params). - **Cameras:** `observation.images.cam1` + `observation.images.cam2`, 128×128, MEAN_STD-normalized. - **Labeling** (all 51 demos are successes, so negatives are synthesized): POSITIVE = last 15% of frames **and** gripper re-opened (release into pot); NEGATIVE = first 55% of frames; the 55–85% transport margin is excluded. Split **by episode** (val = eps [4,14,24,34,44]). - **Deployed with `success_threshold = 0.7`** for release-boundary margin. | metric | value | |---|---| | Val accuracy | **99.49%** | | Val precision / recall / F1 | 0.979 / 0.991 / 0.985 | | Train accuracy (balanced) | 100% | | Confusion (val, thr 0.5) | TP=231, TN=1138, FP=5, FN=2 |   ## 2. Action spec — EE-delta The demos are absolute joints, but the HIL-SERL policy acts in **end-effector delta** space. - **Action = base-frame TCP delta ÷ step_size (0.05 m)**, computed by running **FK on the dataset joints** (identical to what online deploy uses as its per-step reference). The gripper is a discrete class `{0=close, 1=stay, 2=open}`. - The offline buffer stores a 4-dim action (continuous xyz + discrete gripper); the SAC critic slices `actions[:, :-1]` for the continuous part and a separate discrete critic handles the gripper. Hence `output_features["action"].shape = [3]` with `num_discrete_actions = 3`. - Action stats (21,524 frames): p99 ≈ 0.2 in tanh space, `|Δ|>1` = **0.0%** — comfortably inside the `[-1, 1]` range. ### Joint → EE forward kinematics (validated) `joint_to_ee.py` implements UR7e FK directly from the URDF joint origins (placo-free) and was validated against the recorded TCP pose: | subset | pos err median | rot err median | |---|---|---| | near-static (‖q̇‖ **RAM note:** the full offline buffer (`from_lerobot_dataset`) materializes every transition > eagerly and peaks at **~25 GB**. Use ≥48 GB RAM, or subset episodes via > `--dataset.episodes='[...]'`, or lower `offline_buffer_capacity`. ## Rebuilding the offline demo buffer The 54 MB RL demo buffer is not shipped. Rebuild it from the LeRobot dataset with the included script (it converts absolute-joint demos to the EE-delta + reward + done schema): ```bash python build_offline_buffer.py # writes ./banana_rl_lerobot (repo_id theo/banana_in_pot_rl) ``` Then point `dataset.root` in `config/train_hilserl_ur7e.json` at the rebuilt directory (the config ships with a relative `./banana_rl_lerobot`). ## Related repositories - **Dataset:** [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3) - **ACT imitation-learning policy** for the same task: [`Bigenlight/act_banana_in_pot`](https://huggingface.co/Bigenlight/act_banana_in_pot) ## Caveats & limitations - **No true failure episodes.** The classifier's negatives are *early-task* frames (approach/grasp), not genuine failed attempts. It reliably separates "task complete" from "task in progress" but has never seen a real failure of a completed-looking state — expect over-confidence on OOD near-miss end states. **Record a handful of real failure/near-miss episodes early in online HIL-SERL** to harden it. - The excluded 55–85% transport margin means the **decision boundary around the release moment is uncalibrated**; `success_threshold=0.7` adds margin. - **EE safety bounds and reset pose in the config are placeholders** — they MUST be tuned against the real arm before letting the policy move. - Only the classifier head trains (frozen-encoder LeRobot quirk) — fine for this easy visual task, the first thing to revisit if a harder task underperforms. - All validation here is **offline** (config parse → policy/critic build → buffer load → gate). No online RL results are included; that is the robot-only next step.
