Qwen3.5-9B — Four-Head Temporal Ordering (LoRA-merged)
Qwen/Qwen3.5-9B with our LoRA adapter (r=16, alpha=32, targets
q,k,v,o,gate,up,down_proj) merged into the base weights
(PeftModel.merge_and_unload()). This is the backbone for our SNU AI
Challenge submission: given four shuffled video frames and a one-sentence
description, predict their chronological order.
The vision encoder is unmodified from the base model (it was frozen during
training — no LoRA was applied to it, so its weights here are byte-identical
to Qwen/Qwen3.5-9B).
This repo is the backbone only — you also need the rank heads
This model was trained with
four independent rank-classification heads
read from the last non-padding prompt hidden state (see our GitHub repo for
the full source:
code/modeling.py, class
FourIndependentRankHeads). Those
heads are
not part of the standard
AutoModelForImageTextToText weights —
they are a small separate checkpoint bundled in this repo as
direct_head.pt (~200 KB), together with
direct_objective.json
(answer-encoding metadata) and
ablation_manifest.json /
train_args.json (exact training configuration, ablation preset
consistency).
Loading this repo with plain AutoModelForImageTextToText.from_pretrained(...)
gives you a fully functional causal LM + vision backbone, but not an
ordering predictor by itself — you additionally need to load direct_head.pt
onto the last hidden state as documented in our GitHub repo's code/modeling.py
(load_head, four_head_rank_logits) and decode by scoring the 24 valid
permutations in closed form (rank_logits_to_permutation_scores).
Full training/inference code, the LoRA-adapter (unmerged) version, and the
paper describing the method: see the linked GitHub repository.
Why a merged copy, in addition to the LoRA adapter
We also publish the un-merged LoRA adapter (58 MB) in our GitHub repo. This
merged copy exists because folding the adapter into the base weights removes
the extra low-rank matmul branch PEFT adds at every one of the 7 adapted
projections in every decoder layer, which matters most at batch size 1 (our
inference setting): with no batching to amortize the extra kernel launches,
that per-layer LoRA branch is a proportionally larger tax than in typical
large-batch benchmarks. Our 24-view test-time augmentation runs the language
model forward once per permutation, so removing that per-call overhead
compounds over the 24 calls. We have not published a controlled side-by-side
latency number for this specific model/hardware pair; treat the merged copy
as an inference-convenience artifact (also: no peft dependency needed to
load it), not as a benchmarked speedup claim.
Training data / method summary
- Task: SNU AI Challenge frame-ordering (4 frames -> permutation of
1..4)
- Backbone: Qwen3.5-9B, vision encoder frozen throughout
- Adaptation: LoRA r=16/alpha=32/dropout=0, merged here
- Head: 4 independent per-slot rank classifiers over the last prompt hidden
state (LayerNorm -> Dropout -> Linear), decoded by closed-form scoring of
all 24 valid permutations (always a valid output, no autoregressive decode)
- Losses: rank cross-entropy + induced 24-way permutation CE + constrained-
position NLL + pairwise BCE + soft permutation-distance CE + expected
Hamming/Kendall distance + two-view permutation-consistency divergence
(ablation preset
consistency; see ablation_manifest.json)
- Validation (954 held-out rows, seed 42): 58.5% exact match, 71.8% position
accuracy, 83.2% pairwise accuracy (see
eval_metrics.json)
License
Follows the base model's license (Qwen/Qwen3.5-9B).
