GaP — Graph-as-Policy: four-suite evaluation + perception pre-ranking
Code, experiment scripts, execution logs, and per-cell results for reproducing Graph-as-Policy
(GaP) on LIBERO, extending it to all four LIBERO suites (the paper evaluates only the
grocery/Object suite), and for our perception pre-ranking contribution.
→ Progress timeline — what was done when, including the corrections.
→ Trial videos — real renderings of the three failure modes, with per-node attribution.
All numbers below are re-aggregated from the raw cell_result.json files in results/.
Nothing is estimated. Trials killed by quota, connectivity, or out-of-memory are discarded and
re-run — never scored as failures.
Results — baseline vs ours
Paired cells only (both arms have valid data for that task); 3 trials per task per arm.
| Suite | Baseline GaP | + Pre-ranking (ours) | tasks |
|---|
| Object (grocery, the paper's suite) | 28/30 = 93.3% | 29/30 = 96.7% | 10 ✅ |
| Spatial (referring expressions) | 3/30 = 10.0% | 1/30 = 3.3% | 10 ✅ |
| Goal (goal-directed manipulation) | 5/30 = 16.7% | 6/30 = 20.0% | 10 ✅ |
| Long (libero_10, multi-step) | 3/15 = 20.0% | 3/15 = 20.0% | 5 ⏳ |
| TOTAL | 39/105 = 37.1% | 39/105 = 37.1% | 35 |
Perception efficiency — the actual result
| Suite | Baseline calls/trial | Ours calls/trial | Reduction |
|---|
| Object | 14.4 | 2.6 | 5.5× |
| Spatial | 16.3 | 1.9 | 8.4× |
| Goal | 16.7 | 1.6 | 10.2× |
| Long | 6.1 | 1.9 | 3.2× |
Read the two tables together: accuracy is a dead heat (39/105 both arms), at 3–10× fewer
vision-language-model calls. Pre-ranking is an efficiency result, not an accuracy trick.
Where perception is the binding constraint it becomes an accuracy result too — on a
deliberately constrained backend, one task moves 6.7% → 80.0% (30-trial ablation,
Fisher p ≪ 0.001).
Honest caveats
- Long is unfinished (5/10 tasks paired); that row will change.
- Spatial is a regression for our module (10.0% → 3.3%). Cause is diagnosed, not hidden:
graph generation drops the spatial qualifier (
"the bowl between the plate and the ramekin"
→ object_description: "black bowl"), and confident-skip then locks onto the salient bowl
rather than the referred one. Fix implemented and unit-tested in v3_perception/, not yet
benchmarked.
- 3 trials/task, 1 graph sample/task on a free API key with a daily quota. Sanity-scale, not
a full-scale eval.
Where GaP works, and why it stops elsewhere
The traces separate three distinct causes — only one is a real capability ceiling.
① A real limit: the skill library. Opening a drawer, turning a stove knob, pushing a plate
are not pick-and-place. Those tasks ran their full trials with
zero perception aborts and the
longest executions measured —
2680–3297 control steps against ~720 for a success (one cell
reached 18965). The graph was built, perception verified, the arm genuinely tried; the
manipulation primitives do not exist. Visible precisely
because the gates passed.
E.g. videos/goal_t3.mp4: 2320 steps of genuine attempts, checkpoints
failing in order
grasp_pose_above_table → bowl_held → drawer_open_detected.
② A fixable pipeline bug: the referring expression is dropped. See Spatial above.
E.g. videos/spatial_t0.mp4: two identical black bowls, the graph
carrying only
"black bowl", and the run failing at
target_held — the dominant physical failure
outside the grocery suite (
0 occurrences in Object,
56 in Spatial).
③ Graph-generation reliability. Two classes of structurally-valid-but-runtime-fatal graphs
were found and fixed (see below).
Long is the cleanest illustration: task 1 (two objects into a basket, pure pick-and-place)
scores 3/3 in both arms at 1347 control steps — long-horizon is not the problem. Tasks
needing drawers or stoves score 0/3 with long executions. Task 0 cannot be compiled into a valid
graph at all.
Validator fixes (this work)
GaP's validator has 19 structural rules but let two guaranteed-runtime-crash patterns through.
Both now reject at codegen time and trigger a resample instead of silently scoring 0/3.
| Rule | Defect | Symptom before the fix |
|---|
| W9 | parent conditional_edges mapping does not cover every exit a subgraph can emit (exit.success_values ∪ {on_error}) | value 'perception_failed' not in mapping ['found','none'] → 3 trials dead, 0 control steps, scored as a task failure |
| W10 | router_field declared as an empty string (rule S8 only tested is None) | cannot read field '' from output of type dict → 3 trials dead, 0 control steps |
Both are opt-in (strict_exit_coverage=True), enabled only on the codegen path: the executor
raises on error-severity issues, so enforcing them there would reject already-generated graphs
mid-experiment. Patches + unit tests in patches/; archived rejections in logs/w9_hits.log
(10 rejections of the same malformed graph — a systematic model tendency, not a fluke).
A full re-scan confirmed only one completed cell was ever affected by W10 and none of the
published Object/Spatial/Goal numbers are contaminated.
Repository layout
| Path | Contents |
|---|
results/all_cells.json | every cell: trials, successes, avg control steps, wall-clock, park reasons |
logs/gate_suites_main.log | full gate history (CLEAN / PARKED / dirty / infra lines) |
logs/w9_hits.log | archived validator rejections |
patches/ | W9 + W10 validator patches, validate_with_W9_W10.py, unit tests |
v3_perception/ | relation-aware re-ranking + ambiguity guard (14 unit tests, staged) |
scripts/ | per-task benchmark YAMLs, quota-aware resumable gate drivers |
friend_run/ | self-contained bundle to reproduce the whole A/B on another machine |
gap_tables.md / .tex | tables in Markdown and LaTeX |
videos/ | real trial renderings (perception abort, execution limit, grasp-hold failure) |
atomrobot.html | animated skill-graph write-up page |
TIMELINE.md | dated progress timeline + corrections made along the way |
Models
Free reproduction path: gemini-3.1-flash-lite-preview for both perception and codegen via
Google AI Studio's OpenAI-compatible endpoint
(https://generativelanguage.googleapis.com/v1beta/openai).
gemini-3.1-pro-preview is 429-rate-limited on the free tier; flash-lite for both roles still
reproduces ~0.95 on the grocery suite. Reference: arXiv:2607.05369.
Reproducing
See friend_run/README.md — one API key, one command, resumable. Expected cost is a few dollars
of flash-lite calls; ~12 hours on one 24 GB GPU.
API keys are redacted throughout this repository.
