Ngseo/ur5_qwen25vl_act_chunk60

Config

• Base VLM: Qwen/Qwen2.5-VL-3B-Instruct (LoRA rank 32, vision encoder frozen)
• Architecture: VLM hidden -> dual AttentiveLatentHead (A + B) -> concat -> ResNetActionHead, action L1 loss (no V-JEPA, no augmentation)
• Action chunk: 60 (@30fps = 2.0s horizon), action dim 7 (joint_0..5 + gripper)
• State: state-free (observation.state zeroed, state_proj frozen)
• Language conditioning: task description string tokenized by the Qwen processor
• Per-task single camera (input renamed to observation.images.camera1):
  • Point at the red cup -> camera_0
  • Point at the blue cup -> camera_2
  • Pick up the blue die and place in the basket -> camera_0
  • Pick up the red die and place in the basket -> camera_2
  • Pull a tissue out of the box -> camera_1
  • Close the laptop -> camera_1
  • Pick up the shoe and stand it upright -> camera_1
• Training: 100k steps, batch 16, lr 5e-5, cosine schedule

Contents

• checkpoints/0XXXXXX/checkpoint.pt — saved every 10k steps. Each is a torch dict: policy, latent_head, free_latent_head, action_head, optimizer, agg_stats, args.
• ur5_qwen25vl_inference.py — self-contained inference (UR5QwenPolicy).
• ur5_policy_server.py — HTTP policy server (/health, /predict, /reset).
• train_ur5_b_qwen25vl.py — training script.

These checkpoints are not standard LeRobot checkpoints; load them with the included UR5QwenPolicy, which rebuilds the VLM + external heads. Requires the lerobot fork with policies/qwen25vl_act on the path (PYTHONPATH=lerobot/src).

Inference (Python)

Serving (HTTP API)

• GET /health -> policy metadata (cameras, action_dim, per-task camera map).
• POST /predict {"images": {"camera_1": "<base64 JPEG>"}, "task": "..."} -> {"action": [7], "latency_ms": ...}.
• POST /reset -> clears the action queue (call at episode start).