PACTBench evaluates a layered boss-control system across V Rising,
Hollow Knight, and The Binding of Isaac:
L1 converts causal RGB history into a compact, decision-facing belief.
L2 selects among anonymous skill options from the belief and the visible
skill-mechanics descriptions.
This directory is the release unit for code, checkpoints, published metrics,
and reproducibility metadata. Machine-readable publication state lives in
registry/registry.json; prose never overrides the
registry or the referenced metric files.
Current release
The published snapshot uses fight-disjoint splits, the compact belief
interface, balanced-v3 L2 sampling, HP-free-v6 prompts, anonymous option IDs,
Gemma-4-E2B-it at temperature zero, and deterministic sample seed 0.
L1 official evaluation
Game
Test rows
Distance MAE
Angle MAE
Distance-bin accuracy
Angle-bin accuracy
V Rising
8,075
0.2620
11.7599°
0.8320
0.7201
Hollow Knight
8,360
0.4680
3.6527°
0.7261
0.9646
Isaac
1,784
9.0263 px
29.6388°
0.9036
0.8352
Isaac distance is measured in pixels and is not directly comparable to the
engine-unit distance used by the other two games.
L2 natural-ready semantic intervention
Game
Valid paired states
Mean menu size
Menu-aware chance
Description-follow
V Rising
200
4.875
21.93%
67.0%
Hollow Knight
200
5.370
18.78%
79.5%
Isaac
122
2.238
46.04%
83.6%
The 200 source states are fixed before L2 outputs are read. Isaac has 122
valid semantic-swap pairs because the predefined intervention requires at
least two tracker-ready options; singleton menus are excluded before outcome
aggregation. The fixed two-choice controls are reported separately in
docs/EXPERIMENTS.md.
The L1 visual comparison is a comparison of complete training schemes:
PACT-L1 is task-trained end to end, whereas DINOv2-S and VideoMAE-B are frozen
pretrained backbones with trained probes. It is not an isolated backbone claim.
V Rising also includes a separate accepted L1 comparison experiment that trains
DINOv2-S end to end on the same ordered 18,068/8,075 rows and the matched PACT
two-stage recipe. In this configuration it remains less accurate and 48.79%
slower at batch 1 than PACT-L1; the evidence and claim scope are recorded in
docs/EXPERIMENTS.md.
1pactbench/
2├── games/{vrising,hk,isaac}/ # per-game configuration and thin entry points
3├── pipelines/ # shared shell pipelines
4├── pact/ # shared Python implementation
5├── ckpt/ # published L1 checkpoints
6├── results/ # published row-level and aggregate artifacts
7├── registry/ # machine-readable publication snapshot
8├── schemas/ # cross-game schemas
9├── tests/ # regression and provenance tests
10├── data -> ../data # large source/processed data
11└── out -> ../out # rebuildable caches
The decision-quality LLM judge is intentionally run on a separate machine.
Its absence from the local deterministic gate does not change the publication
status of the local L1 and L2 intervention artifacts.