A single 14B model produced by merging three independently fine-tuned checkpoints, each trained on a different target corpus, into one set of weights. The merge is done entirely in parameter space — no joint training on the union of the three corpora.

Merge configuration


Method	TIES (magnitude sparsification + sign-consensus, delta normalization)
Density	ρ = 0.3
Weights	uniform (λ = 1 for each of the three task vectors)
Base / anchor	`Qwen/Qwen2.5-14B-Instruct`
Precision	BF16 throughout
Result	579 tensors, 14.77B parameters

Task vectors are computed relative to the pretrained base, τᵢ = φᵢ − φ₀, and the merged model is φ_merged = φ₀ + TIES({τᵢ}).

Sources

Corpus	Checkpoint
NarrativeQA	`Glow-AI/QWEN2.5-14b_instruct-NQA10_step1-5_batchsize512_best_epoch`
MuSiQue	`Glow-AI/QWEN2.5-14b_instruct-MSQ1000_step1-5_with_N_neg_batchsize256_best_epoch`
BrowseComp-Plus	`1010stonely/Qwen2.5-14B_BCP300_step1-7_N_neg_updated_batchsize1024_trngepoch3`

Merged with lfmerge:

bash

python main.py --method ties \
  --models <nqa> <msq> <bcp> \
  --weight 1 1 1 --density 0.3 0.3 0.3 \
  --base Qwen/Qwen2.5-14B-Instruct \
  --anchor-base

Merge diagnostics

The three task vectors are close to orthogonal, which is the low-interference regime TIES is designed for. Measured on representative weight matrices:

Quantity	Value
pairwise cos(τᵢ, τⱼ)	+0.002 to +0.038
all-three sign agreement	0.24 (chance level for independent signs is 0.25)
‖τ‖/‖φ₀‖	5.5–5.8% (NQA), 6.4–6.6% (MSQ), 7.6–7.7% (BCP)

After merging, all three task vectors are retained with positive alignment:

	cos(φ_merged − φ₀, τᵢ)
NarrativeQA	+0.28 to +0.37
MuSiQue	+0.43 to +0.46
BrowseComp-Plus	+0.57 to +0.62

BrowseComp-Plus is retained most strongly, consistent with it having the largest task-vector norm (it was trained on the most QA pairs). Weight integrity was checked across all 579 tensors: no non-finite values, and all shapes and keys match the base model.

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "yungisimon/Qwen2.5-14B-ties-merge-nqa-msq-bcp"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

Or with vLLM:

vllm serve yungisimon/Qwen2.5-14B-ties-merge-nqa-msq-bcp --dtype bfloat16 --max-model-len 32768

Status and limitations

Not yet evaluated. Downstream accuracy on the three corpora has not been measured for this merge; the diagnostics above are parameter-space measurements only.
Uniform merge weights were used, so the corpus with the largest task-vector norm (BrowseComp-Plus) contributes most. Weighting by inverse task-vector norm is untested.
ρ = 0.3 was chosen because it was the best-performing density in prior merge sweeps on these checkpoints; it was not re-tuned for this three-corpus setting.
The three corpora are treated as pairwise disjoint.

License

Inherits the license of Qwen/Qwen2.5-14B-Instruct (Apache 2.0).