TrajGazeMerge — checkpoints
Stage-1 trajectory encoder + Stage-2 LoRA-finetuned Qwen2.5-VL-7B-Instruct checkpoints for the TrajGazeMerge project — a trajectory-aware visual token selection method that compresses egocentric visual tokens 10× (merge_ratio = 0.9) while keeping the kept tokens task-relevant.
Code:
https://github.com/Jaewoo97/trajgaze (branch
cf-mask-augmented-training).
Paper draft:
docs/NeurIPS_2026_Gaze_hand_Trajectory_Merging_for_Efficient_Egocentric_Video_Understanding.pdf in the source repo.
Best result (one sentence)
TAS-only is best in both 2-dataset (StreamGaze + EgoGazeVQA) and 3-dataset (+ HD-EPIC) setups. Adding ATR / CGM losses hurts mean accuracy; counterfactual-mask diagnosis (see eval_results/) shows these extensions do not change LLM language-prior usage on EgoGazeVQA. See the code repo's docs/paper_narrative_v3.md for the mechanism-by-role reading.
| Setup | Method | StreamGaze | EgoGazeVQA | HD-EPIC | mean |
|---|
| A (2 ds) | TAS-only ★ | 67.49 | 57.77 | — | 62.63 |
| A | TAS+ATR+CGM (FULL) | 61.98 | 59.40 | — | 60.69 |
| A | CGM-only | 62.74 | 55.92 | — | 59.33 |
| B (3 ds) | TAS-only-hdepic ★ | 63.69 | 55.92 | 50.12 | 56.57 |
| B | TAS+ATR-hdepic | 60.65 | 54.76 | 50.66 | 55.35 |
| B | CF-1 (cf-mask aug, early-stopped) | — | — | — | ~54.5 |
| B | CF-3 (cf-mask + shuf aug, early-stopped) | — | — | — | ~50.0 |
Files
stage1/ — Trajectory encoder (Stage-1 pretrained)
| File | Size | Role |
|---|
stage1/E1_combined_AB_TAS.pth | 142 MB | A + B + TAS encoder (Stage-1 supervised). Required as --stage1-ckpt for every Stage-2 / cf-mask run. |
stage2/ — LoRA-finetuned Stage-2 (Qwen-VL + TAS encoder fine-tune)
LoRA: r=16, α=32, target_modules=[q_proj, k_proj, v_proj, o_proj]. Each .pth contains lora_state, encoder_state, and (when applicable) atr_state.
| File | Size | Stage-2 loss | Setup | Notes |
|---|
stage2/E1_combined_TAS_only.pth ★ | 16 GB | CE only | A | Headline (Setup A best). |
stage2/E1_combined_TAS_ATR.pth | 16 GB | CE + ATR | A | Ablation. |
stage2/E1_combined_TAS_ATR_CGM.pth (FULL) | 16 GB | CE + ATR + CGM | A | Ablation; EgoGazeVQA mask_kept sign-flip but StreamGaze regress. |
stage2/E1_combined_CGM_only.pth | 16 GB | CE + CGM | A | Ablation; trained from earlier non-TAS Stage-1. |
stage2/E1_combined_TASonly_hdepic.pth ★ | 16 GB | CE only | B | Setup B best (+ HD-EPIC). |
stage2/E1_combined_TAS_ATR_hdepic.pth | 16 GB | CE + ATR | B | Setup B ablation row. |
stage2/E1_combined_cf1_hdepic.pth | 16 GB | CE + cf-mask margin | B | Direction A (CF-1). Training stopped early at step ~6500. |
stage2/E1_combined_cf3_hdepic.pth | 16 GB | CE + cf-mask + shuf margins | B | Direction A (CF-3). Training stopped early at step ~6500. |
eval_results/ — counterfactual-mask summaries + per-sample parquets
Used to reproduce the 4×2 cf-mask matrix in docs/visual_grounding_diagnosis_v2.md. Each ckpt has per-variant *_mask_summary.json + *_per_sample.parquet for {streamgaze, egovqa}.
How to use
- Clone the code:
git clone https://github.com/Jaewoo97/trajgaze && cd trajgaze && git checkout cf-mask-augmented-training.
- Place the Stage-1 ckpt at
TrajGaze_v2/checkpoints/E1_combined_AB_TAS/best.pth.
- Place each Stage-2 ckpt at
TrajGazeMerge/checkpoints/<run-name>/best.pth matching the file name above (drop the .pth suffix in the dir name; the file inside is always best.pth).
- Run cf-mask diagnostic (no training needed for inference):
1PYTHONPATH=. python -m TrajGazeMerge.eval.counterfactual_mask_eval \
2 --stage1-ckpt TrajGaze_v2/checkpoints/E1_combined_AB_TAS/best.pth \
3 --lora-ckpt TrajGazeMerge/checkpoints/E1_combined_TAS_only/best.pth \
4 --val-dataset streamgaze --tag my_run
See repo README.md for full training commands (including CF-1, CF-3 reproduce commands in §4d).
