A generalized text-conditioned motion tracker (GTP) ONNX model that converts kinematic reference motion trajectories into PD-actuator control targets for the Unitree G1 humanoid (29 DoF, 33 bodies). Trained with the BeyondMimic-style RL pipeline from NVIDIA GEAR's ProtoMotions framework.

This is a tracker, not a text-to-motion generator. Pair it with a kinematic motion source like nvidia/Kimodo-G1-RP-v1 (text → motion diffusion) to get a full text → physics pipeline.

🎬 Verified end-to-end (Kimodo → GTP → MuJoCo, 2026-08-14 on NVIDIA Thor)

Prompt: "a person walking forward with confident strides"

Pipeline:

Kimodo diffusion (100 steps, 11 it/s)         → 8.0s
  → 120 frames @ 30Hz kinematic ref
  → SLERP retime → 199 frames @ 50Hz
  → ProtoMotions GTP ONNX (this model) @ 50Hz  → 1.2s
  → G1 physics @ 1kHz (MuJoCo)                 → 3.2× realtime
  → 100 rendered frames @ 25fps                → ~2s

Result: G1 walked forward, physics-stable, zero-shot on unseen prompt.

Files

File	Purpose
`unified_pipeline.onnx`	The tracker (22 MB) — 8 inputs, 4 outputs (see below)
`unified_pipeline.yaml`	Model+robot+control config (joint order, PD gains, timing)

Interface

Control rate: 50 Hz (control_dt=0.02s) Physics rate: 1000 Hz (physics_dt=0.001s, decimation=20) Anchor body: torso_link (index 16 of 33) Root body: pelvis (index 0)

Inputs (8)

Name	Shape	Kind
`current_anchor_rot`	(1, 4)	quaternion xyzw of torso_link
`current_dof_pos`	(1, 29)	joint positions (rad)
`current_dof_vel`	(1, 29)	joint velocities (rad/s)
`current_root_local_ang_vel`	(1, 3)	pelvis ang-vel in local frame
`historical_processed_actions`	(1, 1, 29)	previous action (PD target rescaled)
`mimic_future_anchor_rot`	(1, 4, 4)	reference torso rot @ steps [1,2,4,8]
`mimic_future_dof_pos`	(1, 4, 29)	reference joints @ steps [1,2,4,8]
`mimic_future_dof_vel`	(1, 4, 29)	reference joint vels @ steps [1,2,4,8]

Future step lookahead: [1, 2, 4, 8] control steps = [20, 40, 80, 160] ms.

Outputs (4)

Name	Shape	Meaning
`actions`	(1, 29)	Raw policy output (before PD conversion)
`joint_pos_targets`	(1, 29)	PD position targets — feed to G1 actuators
`stiffness_targets`	(1, 29)	Per-joint PD stiffness (kp)
`damping_targets`	(1, 29)	Per-joint PD damping (kd)

Control mode: BUILT_IN_PD — the tracker outputs per-step Kp/Kd/target, then the PD controller runs at 1 kHz between control steps.

Joint order (29 DoF)

0-5   left_hip_[pitch,roll,yaw]_joint, left_knee, left_ankle_[pitch,roll]
6-11  right_hip_[pitch,roll,yaw]_joint, right_knee, right_ankle_[pitch,roll]
12-14 waist_[yaw,roll,pitch]_joint
15-21 left_shoulder_[pitch,roll,yaw], left_elbow, left_wrist_[roll,pitch,yaw]
22-28 right_shoulder_[pitch,roll,yaw], right_elbow, right_wrist_[roll,pitch,yaw]

Compatible with the g1_29dof_rev_1_0 URDF/MJCF variant (torso waist_yaw+roll+pitch, wrists roll+pitch+yaw, no hand fingers — rubber hand end-effectors).

MJCF reference: mjcf/g1_holo_compat.xml

Body order (33 bodies)

0  pelvis
1  head
2-7  left leg: [hip_pitch, hip_roll, hip_yaw, knee, ankle_pitch, ankle_roll]_link
8-13 right leg (mirror)
14   waist_yaw_link
15   waist_roll_link
16   torso_link       ← anchor
17-23 left arm: [shoulder_pitch, shoulder_roll, shoulder_yaw, elbow, wrist_roll, wrist_pitch, wrist_yaw]_link
24   left_rubber_hand
25-31 right arm (mirror)
32   right_rubber_hand

Quickstart (Python)

python

import numpy as np, onnxruntime as ort, yaml

sess = ort.InferenceSession("unified_pipeline.onnx", providers=["CPUExecutionProvider"])
cfg  = yaml.safe_load(open("unified_pipeline.yaml"))

# Build observation from your robot state + kinematic reference motion
obs = {
    "current_anchor_rot":           np.zeros((1, 4),      dtype=np.float32),   # torso quat
    "current_dof_pos":              np.zeros((1, 29),     dtype=np.float32),   # rad
    "current_dof_vel":              np.zeros((1, 29),     dtype=np.float32),   # rad/s
    "current_root_local_ang_vel":   np.zeros((1, 3),      dtype=np.float32),
    "historical_processed_actions": np.zeros((1, 1, 29),  dtype=np.float32),
    "mimic_future_anchor_rot":      np.zeros((1, 4, 4),   dtype=np.float32),   # 4 lookahead steps
    "mimic_future_dof_pos":         np.zeros((1, 4, 29),  dtype=np.float32),
    "mimic_future_dof_vel":         np.zeros((1, 4, 29),  dtype=np.float32),
}

actions, joint_targets, stiff, damp = sess.run(
    ["actions", "joint_pos_targets", "stiffness_targets", "damping_targets"],
    obs,
)
# → feed joint_targets/stiff/damp to G1 PD controller at 50 Hz

Text → Motion → Physics pipeline (with Kimodo)

python

# 1) Generate kinematic reference from text (nvidia/Kimodo-G1-RP-v1)
from diffusers import DiffusionPipeline  # or Kimodo's native API
motion = kimodo.generate("a person walking forward with confident strides")
# motion: (120, 3+29) root_pos + joint_pos at 30 Hz

# 2) Retime to 50 Hz (SLERP for rotations, linear for positions)
motion_50hz = slerp_retime(motion, from_fps=30, to_fps=50)   # → (199, 32)

# 3) Roll out through GTP tracker in MuJoCo (see full script link below)
for step in range(len(motion_50hz)):
    obs = build_obs(mj_data, motion_50hz, step, lookahead=[1,2,4,8])
    _, joint_targets, kp, kd = sess.run(None, obs)
    apply_pd(mj_data, joint_targets, kp, kd)
    for _ in range(20):  # 20× physics substeps → 1 kHz
        mujoco.mj_step(mj_model, mj_data)

Training

Framework: ProtoMotions (NVIDIA GEAR)
Method: BeyondMimic-style RL tracking (imitation + PD residual)
Simulator: IsaacLab / MuJoCo Warp
Checkpoint: exps/exp-20260306_110148/last.ckpt
Reference paper: BeyondMimic — arXiv:2408.07295

Integration status

All three integrations have landed on strands-labs/robots main:

✅ MuJoCo — verified end-to-end via strands-robots (walk, boxing, squats, jumping-jacks generated on NVIDIA Thor)
✅ strands_robots.policies.kimodo.KimodoPolicy — first-class text→motion provider (#2270, merged 2026-08-15)
✅ strands_robots.policies.protomotions.ProtoMotionsPolicy — native ONNX GTP tracker consuming this model (#2287, merged 2026-08-16)
✅ Kimodo → Unitree G1 action-key bridge — pure key rename between the KIMODO_G1_JOINTS vocabulary and lerobot's unitree_sdk2 driver (#2279, merged 2026-08-16). RTPS/DDS transport is already available via use_rtps (pip-only cyclonedds, no ROS 2 install needed).

Text → physics with strands_robots, one call:

bash

pip install "strands-robots[kimodo,protomotions,sim-mujoco]"
export STRANDS_TRUST_REMOTE_CODE=1     # NVIDIA Open Model License gate
export MUJOCO_GL=egl                   # headless render on Jetson/Thor

python

from strands_robots import Robot, create_policy

sim = Robot("g1", mesh=False)
sim.add_camera("cinematic", position=[3, 0, 1.2], target=[0, 0, 0.9])
sim.start_recording(repo_id="my/kimodo-g1", root="./data", fps=25)

kimodo = create_policy(
    "kimodo",
    diffusion_steps=100,
    guidance_scale=7.5,
    device="cuda",
)
tracker = create_policy(
    "protomotions",
    onnx_path="cagataydev/protomotions-gtp-unitree-g1/unified_pipeline.onnx",
    yaml_path="cagataydev/protomotions-gtp-unitree-g1/unified_pipeline.yaml",
)

sim.run_policy(
    robot_name="g1",
    policy=kimodo,
    tracker=tracker,                       # feeds Kimodo's qpos into GTP → PD targets
    instruction="a person walking forward with confident strides",
    n_steps=200,
    control_frequency=50,                  # ProtoMotions native rate
    n_episodes=10,
    reset_between=True,
)
sim.stop_recording()

Full working example: examples/kimodo/kimodo_g1_walking.py + kimodo_g1_dataset_headcam.py (head-mounted first-person camera → LeRobot v3 parquet dataset).

Real Unitree G1 hardware (same policy, same API — just swap the sim for a hardware bridge):

python

from strands_robots.policies.kimodo.hardware import kimodo_action_to_lerobot_g1

# Kimodo emits 29-DoF whole-body qpos targets in KIMODO_G1_JOINTS ordering.
# The bridge is a pure key rename to lerobot_unitree's driver vocabulary:
lerobot_action = kimodo_action_to_lerobot_g1(kimodo_action)

Wire that action dict into any DDS transport — use_rtps speaks the same RTPS wire that Unitree's unitree_sdk2 uses, so an advertise + publish cycle on rt/lowcmd reaches the real robot without a ROS 2 install. Kill-switch, torque clamps, and damp-mode watchdog live at the driver layer.

Chained multi-prompt long-horizon runs are stable end-to-end after PR #2353 (each new prompt eases off the last commanded pose so no pop between segments) and PR #2284 (per-episode seeds actually reach the sampler). See the 10k-episode dataset generated from this stack.

Citation

If you use this tracker, please cite ProtoMotions + BeyondMimic:

bibtex

@article{beyondmimic,
  title={BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion},
  year={2024},
  eprint={2408.07295},
  archivePrefix={arXiv},
}
@misc{protomotions,
  author={NVIDIA GEAR},
  title={ProtoMotions},
  howpublished={\url{https://github.com/NVlabs/ProtoMotions}},
}

License

Apache-2.0 (matches ProtoMotions upstream). Robot assets (Unitree G1 MJCF) subject to Unitree's own license.

Contact

Maintainer: @cagataycali
Uploaded from NVIDIA Thor (Jetson Thor Dev Kit) as part of the strands-labs/robots research stack.