HIL-SERL offline prep bundle — Put the right banana in the pot (UR7e)
This repository is the offline-prepared base for HIL-SERL (Human-in-the-Loop Sample-Efficient
Robotic reinforcement Learning) on the "put the right banana in the pot" task with a
Universal Robots UR7e. It contains everything up to online RL — the vision reward
classifier, the SAC/RLPD training config, the UR7e kinematics (URDF + FK), the offline demo
buffer builder, and the runbook. The online phase itself is robot-only and is not included
here (it requires the physical arm plus ur_rtde + placo + a gRPC actor/learner).
Offline demo buffer is NOT shipped (it is ~54 MB of video/state transitions). Rebuild it
from the dataset with build_offline_buffer.py — see Rebuilding the demo buffer.
1. Reward / success classifier
A vision-based binary success detector trained offline, in the exact LeRobot
reward_classifier format so it drops straight into HIL-SERL.
Encoder: frozen lerobot/resnet10 (CNN), per-camera SpatialLearnedEmbeddings pool +
Linear→LayerNorm→Tanh. Following the official LeRobot behavior, the encoder runs under
no_grad so only the classifier head trains (~2.36 M trainable params).
Labeling (all 51 demos are successes, so negatives are synthesized): POSITIVE = last 15%
of frames and gripper re-opened (release into pot); NEGATIVE = first 55% of frames; the
55–85% transport margin is excluded. Split by episode (val = eps [4,14,24,34,44]).
Deployed with success_threshold = 0.7 for release-boundary margin.
metric
value
Val accuracy
99.49%
Val precision / recall / F1
0.979 / 0.991 / 0.985
Train accuracy (balanced)
100%
Confusion (val, thr 0.5)
TP=231, TN=1138, FP=5, FN=2
training curves
confusion matrix
2. Action spec — EE-delta
The demos are absolute joints, but the HIL-SERL policy acts in end-effector delta space.
Action = base-frame TCP delta ÷ step_size (0.05 m), computed by running FK on the dataset
joints (identical to what online deploy uses as its per-step reference). The gripper is a
discrete class {0=close, 1=stay, 2=open}.
The offline buffer stores a 4-dim action (continuous xyz + discrete gripper); the SAC critic
slices actions[:, :-1] for the continuous part and a separate discrete critic handles the
gripper. Hence output_features["action"].shape = [3] with num_discrete_actions = 3.
Action stats (21,524 frames): p99 ≈ 0.2 in tanh space, |Δ|>1 = 0.0% — comfortably inside
the [-1, 1] range.
Joint → EE forward kinematics (validated)
joint_to_ee.py implements UR7e FK directly from the URDF joint origins (placo-free) and was
validated against the recorded TCP pose:
subset
pos err median
rot err median
near-static (‖q̇‖<0.02, n=1884)
0.85 mm
0.16°
all samples (n=42,833)
28.0 mm
—
Sub-mm / sub-0.2° error while static confirms the kinematic chain is accurate; the larger
moving-sample error is timing jitter between the two async logging streams (joint vs. TCP),
not an FK error. Online deploy IK uses placo (Pinocchio), frame tool0.
FK validation
3. Training config (SAC + RLPD)
config/train_hilserl_ur7e.json is the shared TrainRLServerPipelineConfig for the learner and
actor, fully consistent with the offline buffer:
action[3] (continuous xyz; gripper via discrete head)
env (gym_manipulator)
resize_size=[128,128], EE step_sizes=0.05, IK urdf=ur7e.urdf/tool0, reward_classifier thr 0.7, robot/teleop = null
The learner setup was validated end-to-end offline: python -m lerobot.rl.learner builds the SAC
policy (2.76 M trainable / 7.67 M total params), loads the offline demo buffer via
ReplayBuffer.from_lerobot_dataset, starts its gRPC server, and idles at the online-buffer gate
(len(online_buffer) < online_step_before_learning=100) waiting for the actor — which is exactly
the "ready for online RL" state.
4. The online phase (robot-only, NOT in this repo)
See HILSERL_RUNBOOK.md for the exact sequence. In short, on the machine wired to the arm:
Install the hardware/transport deps that the offline env intentionally omits: grpcio
(py3.12 wheel), placo (IK), ur_rtde (UR I/O), lerobot[hilserl].
Fill env.robot / env.teleop in the config (UR7e IP + motors + cameras; gamepad teleop).
Tune the workspace placeholders against the real arm: end_effector_bounds,
fixed_reset_joint_positions, episode length. Keep end_effector_step_sizes = 0.05 — the
offline action = TCP-delta ÷ 0.05, so changing it desyncs the demo actions.
Start the learner (terminal 1) and actor (terminal 2) with the same config. The learner
idles at the gate until the actor supplies ≥100 online transitions, then SAC updates begin with
the 50/50 RLPD mix.
Human interventions: gamepad trigger / space to take over and give corrective demos; taper
the intervention rate as the policy improves.
RAM note: the full offline buffer (from_lerobot_dataset) materializes every transition
eagerly and peaks at ~25 GB. Use ≥48 GB RAM, or subset episodes via
--dataset.episodes='[...]', or lower offline_buffer_capacity.
Rebuilding the offline demo buffer
The 54 MB RL demo buffer is not shipped. Rebuild it from the LeRobot dataset with the included
script (it converts absolute-joint demos to the EE-delta + reward + done schema):
No true failure episodes. The classifier's negatives are early-task frames (approach/grasp),
not genuine failed attempts. It reliably separates "task complete" from "task in progress" but has
never seen a real failure of a completed-looking state — expect over-confidence on OOD near-miss
end states. Record a handful of real failure/near-miss episodes early in online HIL-SERL to
harden it.
The excluded 55–85% transport margin means the decision boundary around the release moment is
uncalibrated; success_threshold=0.7 adds margin.
EE safety bounds and reset pose in the config are placeholders — they MUST be tuned against
the real arm before letting the policy move.
Only the classifier head trains (frozen-encoder LeRobot quirk) — fine for this easy visual task,
the first thing to revisit if a harder task underperforms.
All validation here is offline (config parse → policy/critic build → buffer load → gate). No
online RL results are included; that is the robot-only next step.
