Views
No views yet
high_noise_model and
low_noise_model). Their weights differ substantially in magnitude (mean
‖W_high − W_low‖_F / ‖W_low‖_F ≈ 0.46) but the difference is highly low-rank — rank-99 of
Δ is ~10% of min(m, n) per layer. This repo stores:base.<layer_key> — W_low (the low-noise expert; also serves as the shared base)delta.<layer_key>.{U,S,Vt} — SVD of (W_high − W_low) at rank q=min(min_dim, 512)delta_raw.<layer_key> — raw difference for 1-D / tiny tensors (biases, norms)1from huggingface_hub import hf_hub_download
2from safetensors.torch import load_file
3import json
4
5state = load_file(hf_hub_download("gueraf/wan2.2-14b-svd-merge", "wan22_14b_svd_merge.safetensors"))
6meta = json.loads(open(hf_hub_download("gueraf/wan2.2-14b-svd-merge", "meta.json")).read())
7
8# At runtime, reconstruct W_high at chosen rank r per layer:
9key = "blocks.0.self_attn.q.weight"
10W_low = state[f"base.{key}"]
11U, S, Vt = state[f"delta.{key}.U"], state[f"delta.{key}.S"], state[f"delta.{key}.Vt"]
12r = int(0.10 * meta[key]["min_dim"]) # 10% of min_dim ≈ 99% of Δ energy
13W_high_approx = W_low + (U[:, :r] * S[:r]) @ Vt[:r, :]Wan-AI/Wan2.2-T2V-A14B (high_noise_model/wan2.2-t2v-14b-highnoise.pth,
low_noise_model/wan2.2-t2v-14b-lownoise.pth), fp32.torch.svd_lowrank(q=512, niter=6) —
q=512 captures the full useful spectrum since rank-99 of Δ is ~10% of min_dim (~358 for 14B).