Dual-head retrieval + generation adapter on Qwen/Qwen3.5-4B. Toggle the LoRA at inference: on for ColBERT-style late-interaction retrieval, off for autoregressive generation on the base model.
Hydra starts from the observation that a LoRA adapter trained for retrieval leaves the base model's weights intact by construction: disabling the adapter recovers the original generation head bit-for-bit. The contribution of the paper is identifying three engineering requirements that make this usable in practice — attention-mode restoration (causal → bidirectional toggle on the full-attention layers), lm_head preservation under weight tying and DDP gradient synchronisation, and KV-cache-aware generation — and showing that, once those are addressed, one VLM instance can serve both ColBERT-style late-interaction document retrieval and autoregressive generation without any generation training and with peak VRAM approximately half that of running the two models separately.
This 4B release is a larger-scale instantiation of the same mechanism, scaled to the full ~760K multilingual data mix.
All 22 tasks: nDCG@5 0.7215, nDCG@10 0.7410. Per-task JSONs in results/evals/.
Generation equivalence
When the LoRA is disabled, the Hydra model is the vanilla Qwen/Qwen3.5-4B. The audit in scripts/test_gen_equivalence_4b.py checks three invariants and a per-tensor state-dict comparison:
adapter_config.json has no gen-path modules (lm_head, embed_tokens) in modules_to_save (only the retrieval-side custom_text_proj is saved as a full module)
adapter_model.safetensors contains only LoRA A/B pairs and the custom_text_proj weight; nothing touches lm_head or embed_tokens
Every language_model weight tensor in the Hydra stack is byte-for-byte identical to the corresponding weight in a freshly-loaded base model (426/426 tensors match bitwise)
The left panel below shows the three invariants (all pass). The right panel shows the state-dict comparison across all language-model weight tensors.
Generation equivalence
Mode-switching VRAM
The dual-head design means one process holds one set of weights, toggling the LoRA in place. A conventional setup needs two separate models in memory for the same retrieve-then-generate flow. Both configurations are measured on the same hardware with the same inputs (scripts/bench_mode_switch_vram_4b.py).
Full optimizer, scheduler, and RNG state are in checkpoints/step-{500,1000,1500,2000,2500,3000,3020}/. Resume with the same WORLD_SIZE as the original run (one per-rank rng_state_*.pth was written).
The baseline/ folder holds a matched single-head retrieval-only ColQwen3.5-4B trained on the identical recipe (LoRA r=32 / alpha=32 / dropout=0.05, 3020 steps of DocMatix, seed 123). It exists solely as the paired control for the retrieval claim in results/retrieval/vs_baseline/: with no architecture and no hyperparameter changes, the dual-head modification adds no measurable retrieval cost.
Comparison
All 22 tasks
Hydra-4B (dual-head) mean NDCG@5
0.7215
Baseline (single-head) mean NDCG@5
0.7210
Mean Δ
+0.0005
Paired t-test
p = 0.89
Wilcoxon
p = 1.00
Cohen's d (paired)
0.03
95% CI on Δ
[−0.006, +0.007]
Wins / Ties / Losses
9 / 2 / 11
Contents: final adapter (step-3020), 5 resume checkpoints (step-1500..3020), 22 per-task Vidore JSONs under baseline/results/eval__colqwen_baseline_r32_seed123/, train.log, and the training script used to produce it.
