C2 task-sum lambda=0.8 — merge ablation (NOT the recommended checkpoint)
This is a negative result published for completeness. It performs worse than
yungisimon/Qwen2.5-14B-ties-merge-nqa-msq-bcp, which is the checkpoint to use. This one exists to document what was
tried and ruled out.
Merges three 14B MEMORY models — each fine-tuned on a different corpus (NarrativeQA,
MuSiQue, BrowseComp-Plus) — into one set of weights, entirely in parameter space.
Recipe
1python main.py \
2 --models <nqa> <msq> <bcp> \
3 --base Qwen/Qwen2.5-14B-Instruct \
4 --method task --weight 0.8 0.8 0.8 --density 1 1 1 --raw-weights --anchor-base
Task vectors are computed against the pretrained base, tau_i = phi_i - phi_0.
| |
|---|
| alpha = <D_merged, tau_i>/||tau_i||^2 (nqa / msq / bcp) | 0.86 / 0.85 / 0.84 |
| ||D_merged|| / ||phi_0|| | 8.4% |
| single-corpus fine-tune, for reference | 4.8–7.0% |
alpha = 1.0 would mean a corpus's fine-tuning survived the merge at full strength.
Measured result
Single-turn knowledge recall: the MEMORY model is asked corpus questions directly, with no
multi-turn protocol, judged against gold answers by Qwen2.5-32B-Instruct. n=80 per corpus,
restricted to the subsets each model was actually trained on.
The reference figures are from the same probe run as this checkpoint, because cross-run
noise is about +/-5pp at n=80 (the reference measured 21.2 / 16.2 / 25.0 on BCP across three
runs). Differences smaller than ~5pp are not resolvable at this sample size.
Sum at reduced strength, chosen to keep the perturbation near that of a single fine-tune. Still far below the averaged TIES reference.
Why these all failed
The three task vectors are near-orthogonal in aggregate (pairwise cosine 0.002–0.038), which
suggests summing them rather than averaging, since orthogonal updates should not interfere.
That reasoning is wrong: 75.4% of coordinates carry a sign conflict between the three
vectors. Global orthogonality does not imply coordinate-wise compatibility. TIES's magnitude
trimming and sign-consensus filtering are therefore doing necessary conflict resolution, not
merely attenuating the task vectors — so recipes that preserve more of each task vector
(higher alpha) consistently do worse. Six candidates across four families (pure sum, sparse
sum, sub-0.3 sparsity, non-uniform weighting) all lost to plain TIES rho=0.3 with uniform
averaging.
Caveats
- The probe asks the full question in a single turn, whereas the MEMORY model is trained
on decomposed, self-contained reflection pairs and is used at inference on decomposed
sub-queries. Absolute numbers are therefore well below published protocol accuracy and are
not comparable to it; the probe is a relative screen of knowledge accessibility.
- Judging is a single greedy pass of one local model, not human-validated.
- A NarrativeQA column was collected but discarded: the NQA specialist scored 0% on its own
corpus, so that probe measures prompt-style mismatch rather than knowledge.
License
Inherits the license of Qwen/Qwen2.5-14B-Instruct (Apache 2.0).
