EK100 Action Anticipation Package
This package contains the pieces needed by a collaborator who already has the
EK100 mean-pooled context features and context Pi3 trajectories.
Included:
- V-JEPA 2.1 full-token context features for train, val, and test.
- Pi3 future-action trajectories for train, val, and test.
- Train/validation/test action-anticipation manifests.
- Full-token extraction shard manifests for reconstructing shard membership.
Not included:
- Mean-pooled context features.
- Context Pi3 trajectories from
[t-2, t-1].
- Raw videos.
Layout After Extraction
The paths inside the tar files are relative to the original EK100 extracted root:
1features/train/full_tokens/*.pt
2features/val/full_tokens/*.pt
3features/test/full_tokens/*.pt
4
5pi3_future_action/atomic_train/pi3_trajectory_controls_atomic/m16/*.pt
6pi3_future_action/atomic_dev/pi3_trajectory_controls_atomic/m16/*.pt
7pi3_future_action/test/pi3_trajectory_controls_atomic/m16/*.pt
8
9planning/atomic_train/manifests/segment_manifest.csv
10planning/atomic_dev/manifests/segment_manifest.csv
11planning/test/manifests/segment_manifest.csv
12
13manifests/fulltoken_shards/{train,val,test}/vjepa_shard_*.csv
Timing Convention
For each EK100 segment i, the action anticipation context is the clip before
the action:
- context interval:
[t_i - 2s, t_i - 1s]
- anticipation gap:
1s
- target action interval:
[action_start_sec, action_end_sec]
The full-token V-JEPA features are extracted from the same context interval used
for the mean-pooled cache. The future Pi3 trajectories are sampled from the
target action segment, using the same 16-knot center/owned-interval style used by
the current predictor experiments.
Tensor Formats
Full-token feature .pt files are dictionaries. The main tensor is:
Future Pi3 trajectory .pt files are dictionaries. The main trajectory tensor is:
The trajectory files also store query timestamps, requested frame indices, gravity
anchor metadata, video metadata, and Pi3 provenance.
Counts
1train manifest rows: 66,178
2val manifest rows: 9,595
3test manifest rows: 13,061
4
5full-token train files: 66,178
6full-token val files: 9,587
7full-token test files: 13,061
8
9future Pi3 train files: 66,178
10future Pi3 val files: 9,593
11future Pi3 test files: 13,061
Known missing validation files:
- full tokens: 8 validation segments
- future Pi3:
P30_08_154, P30_08_29
The missing future Pi3 entries are from corrupt/decode-failing frames in
P30_08.MP4.
Current Local Predictor Results
These are local EK100 validation mean-class Recall@5 numbers, matching the
benchmark metric style. The trajectory-conditioned row uses context Pi3 plus
oracle future-action Pi3, so it is an oracle-future-trajectory validation setting.
1No-traj predictor: Verb 17.74 / Noun 17.51 / Action 5.37 / VN avg 17.63
2Oracle future Pi3 predictor Verb 19.57 / Noun 19.26 / Action 6.27 / VN avg 19.42
Both predictor rows initialize from the atomic causal text bridge checkpoint and
use its original atomic CamFormer trajectory encoder. The separately trained
EK100 CamFormer alignment diagnostic checkpoint is not used in those two numbers.
