LoRA adapter for Qwen/Qwen3.6-27B (qwen3_5 architecture,
hybrid GatedDeltaNet + full-attention), fine-tuned to re-time an already-known ordered list of
action segments: given the video plus the segments' text (no timing), it predicts the correct
MM:SS.d - MM:SS.d for each, in order.
This is not an error-correction model trained on (correct caption, correct timestamp) pairs --
feeding the true timestamp back in as input would let it learn to just copy it through without
looking at the video. Instead its training target strips timestamps out of the input entirely
(stage_2_2_timestamp_only_with_caption/scripts/build_caption_conditioned_manifest.py), so there
is nothing to copy and the only way to answer is real visual grounding. It is the second stage of
a two-stage pipeline: a captioning model (e.g.
bi199797/VR-egodex-narration-qwen36-27b-lora-vision12-lr1e5)
supplies the what, this model supplies the when -- conditioning on the given text takes the
in-loop timestamp score from 0.6048 (the
video-only boundary sibling,
no text at all) to 0.6868 on the same eval sample.
Training code: VR-finetune-VLM, stage_2_2_timestamp_only_with_caption/.
Adapter config
Base weights
bf16 (full precision, not quantized)
Adapter
LoRA, rsLoRA scaling
lora_r
32
lora_alpha
64
lora_dropout
0.05
Target modules
language-model attention/MLP only (q/k/v/o_proj, gate/up/down_proj) -- no vision-tower adaptation
Trainable params
~200.3M / 27.6B (0.73%)
Training
Data
19,672 EgoDex episodes (stage_2_2_timestamp_only_with_caption/data/manifests/train.jsonl) -- video + the episode's own ordered, timestamp-free action-segment text folded into the prompt as episode context, target is the ground-truth MM:SS.d - MM:SS.d for each listed segment
Prompt
stage1/prompts/boundary_with_caption/v1.txt -- "find the correct start and end time for each listed action segment"
Epochs
2 (1,230 steps)
Batch size
1 x 16 grad-accum = 16 effective
Learning rate
1e-4, cosine schedule, warmup 3%
Weight decay
0.01
Sampling
fps=2.0, max_frames=128, max_pixels=272384/frame
Max sequence length
40,960
Precision
bf16
Cluster
2x NVIDIA H100 80GB (device_map=auto)
Eval (stage0.eval timestamp axis: cue-boundary IoU only, no text)
In-loop eval logged during training (stage2/train.py's periodic callback, 24-episode sample
of stage_2_2_timestamp_only_with_caption/data/manifests/val.jsonl) -- not yet the full
300/549-episode generation sweep the narration cards report.
step
timestamp_score
eval_loss
100
0.6664
--
600
0.6587
--
1000
0.6903
--
1100 (peak)
0.6983
--
1230 (final)
0.6868
0.1259
Consistently 0.65-0.70 across the whole run -- well above the video-only boundary sibling's
0.52-0.60 band on the same schedule, which is the intended effect of conditioning on the
given caption text rather than localizing blind.
Not yet measured end to end: whether re-timing a predicted caption (from a captioning
model that gets some segments wrong) holds up as well as re-timing the ground-truth segments
this checkpoint was trained and evaluated against here. A two-stage pipeline run
(stage_2_2_timestamp_only_with_caption/scripts/run_pipeline_deakin.sbatch) chaining this
checkpoint after a captioning model's own predictions, scored on the full test/val splits, is
in progress.
Usage
This model expects the video plus the ordered segment text folded into the prompt as
episode context -- it is not a plain video-to-timestamp model (see the
boundary sibling for that).
Build the prompt with the system prompt at stage1/prompts/boundary_with_caption/v1.txt, plus
episode context ending in:
## Predicted action segments (in order; find the correct start/end time for each)
1. [left hand] reach toward the small object | [right hand] rest on the table | [ego] stay still
2. [left hand] grasp and lift the small object | [right hand] hold the container steady | [ego] stay still
...
The model returns exactly one MM:SS.d - MM:SS.d line per listed segment, in the same order,
first line starting at 00:00.0, last line ending at the video's final frame.
