You can deploy this model and start earning money today!

chomeed/
v_state_mimicgen_coffee_d0

Name: v_state_mimicgen_coffee_d0
Availability: InStock

pytorch

reinforcement-learning

Views

No views yet

State-value models V(s) for mimicgen coffee_d0

Two state-only value functions predicting the discounted return (gamma = 0.99) from the 66-d proprioceptive state. Trained without any access to a learned Q critic — only success/failure trajectories and the datasets' own human annotations. The armF2 critic is used as an evaluation reference only.

	`v_td`	`v_relabel_hinge`
pearson vs armF2 Q	0.854	0.888
spearman vs Q	0.870	0.857
pearson vs Q, failure chunks	0.671	0.725
pearson vs Q, ±40 frames of the annotated failure	0.609	0.632
MAE vs Q	0.1077	0.0772
pearson vs true return G	0.839	0.812
MAE vs G	0.0827	0.0882

Split B holdout (episodes 80-99 of each rollout set), stride-8 chunk grid, n = 1577. Three seeds each; the table scores the mean of the three predictions.

The two models

v_td — plain TD regression. Target y = r + (1-done) * gamma^8 * V_target(s_{t+8}), smooth-L1, hard target sync every 200 steps. Reward is 0 everywhere except the terminal frame of a success episode, so once the +8 lookahead passes the end the target is the exact closed-form return.

v_relabel_hinge — the same, plus two ingredients built from the annotations:

Hindsight-relabelled failure targets. On a failure episode the ordinary target is 0 at every frame and says nothing about when it went wrong. Using meta/lerobot_annotations.json, for t < t_fail the target becomes the matched success stage's return at the same stage fraction u = t / t_fail, and 0 after. Blended y = 0.75 * y_TD + 0.25 * y_relabel. Mode -> stage: failed to grasp pod -> good grasp, pod misaligned -> good insertion.
A margin hinge on matched pairs. relu(0.1 - (V(s_success) - V(s_failure))) over 5851 matched (success stage-end, failure-mode) state pairs, anchors jittered +-5 frames. A margin matters: the unbounded Bradley-Terry form softplus(-(V_succ - V_fail)) blows the value range from ~1.2 to 5.2 and drops pearson-vs-Q to 0.438.

Usage

python

import torch, numpy as np
from modeling_v import VNet, scale_state

net = VNet(66)
net.load_state_dict(torch.load("weights/v_relabel_hinge_seed0.pt", map_location="cpu"))
net.eval()

x = torch.from_numpy(scale_state(raw_state_66d))      # scaling is REQUIRED
with torch.no_grad():
    value = net(x)

scale_state applies the armF2 critic's own q01/q99 normalisation (shipped as state_scaler.json) and clips to +-5. Feeding raw states will not work.

For the reported numbers, average the three seeds' predictions.

Known limitations

Not usable for advantage. gamma^8 * V(s') - V(s) correlates -0.111 with the true advantage Q - E_a[Q] and its sign agrees 45.2% of the time. The true advantage has sd 0.0047 against this quantity's 0.055 — at the prior's action spread the action explains 0.15% of Var[Q]. Use V as a baseline or a shaping potential, not as a source of action rankings.
Fit on prior-policy rollouts only (mtdit_flow_60k). Expect degradation as a policy moves off that distribution.
The relabelling needs annotated failure frames, which fresh rollouts do not have. Retraining online requires predicted failure anchors.
p_head in the checkpoints is untrained; ignore it.

Data

chomeed/mimicgen_coffee_d0_224x224_mtdit_flow_60k_{success,failure} (100 episodes each, 20 fps, 66-d state), including the meta/lerobot_annotations.json markers the relabelling depends on.

chomeed/v_state_mimicgen_coffee_d0