This repository contains factor-only Sparse Weight Decomposition (SWD)
checkpoints used in our replacement-fidelity and circuit-extraction
experiments. It does not redistribute any base model. Load the corresponding
base model first, then apply one checkpoint with swd_loader.py.
Each replaced matrix is represented as
output = input @ read @ write + bias
The intermediate coordinates are the SWD bottleneck units used for circuit
scoring and ablation. s=0.5 and s=0.75 mean that 50% and 75% of all entries
across the two factors are zero, respectively.
Included Checkpoints
Base model
Replacement
Setting
Data used
CE delta vs dense
GPT-2 Small
Layer 8 mlp.c_proj
s=0.5
16,384 tokens
0.000889
GPT-2 Small
Layer 8 mlp.c_proj
s=0.75
16,384 tokens
0.008292
Qwen2.5-0.5B
Layer 12 mlp.down_proj
s=0.5
1,024 tokens
0.001222
Qwen2.5-0.5B
Layer 12 mlp.down_proj
s=0.75
1,048,576 tokens
0.005010
Qwen2.5-1.5B
Layer 14 mlp.down_proj
s=0.5
1,024 tokens
0.000733
Qwen2.5-1.5B
Layer 14 mlp.down_proj
s=0.75
1,048,576 tokens
0.001222
Qwen3.5-27B
Layer 31 mlp.down_proj
s=0.5
2,048 tokens
-0.000427
Qwen3.5-27B
Layer 31 mlp.down_proj
s=0.75
1,048,576 tokens
-0.000448
GPT-2 Small
Layer 8 complete MLP
s=0.5
16,384 tokens
0.004003
GPT-2 Small
Layer 8 complete MLP
s=0.75
1,048,576 tokens
0.015521
GPT-2 Small
All 48 transformer-block linear projections
fixed-support SWD-FT
20,578,304 tokens
0.151263*
* The full-model value uses its full-model stress evaluation and should not
be numerically compared with the single-matrix unified CE rows.
The Qwen2.5-3B checkpoints are distributed separately at
veri-safe/SWD-Qwen2.5-3B
because the upstream model uses the Qwen Research License.
mode="factorized" installs SWDLinear, exposing
component_activations(inputs). Use mode="folded" to write read @ write
back into the original dense module for conventional inference.
The base model must be fully materialized before applying a checkpoint. For
large models loaded with a device map, each replacement is moved to the device
and dtype of its target module.
Format
Every checkpoint directory contains:
text
1model.safetensors # factor tensors only; no pickle
2config.json # base model, module paths, shapes, sparsity, and token exposure
3provenance.json # source hashes, conversion rule, and validation result
All public factors follow [input, rank] @ [rank, output], independent of the
source framework's dense-weight layout. Qwen2.5 source factors are transposed
into this convention; Qwen3.5 feature shards are concatenated along the rank
dimension without changing dtype or values. Full-model GPT-2 checkpoint biases
are included because they belong to the fixed-support fine-tuned checkpoint.
RELEASE_MANIFEST.csv is the machine-readable checkpoint index. Each
checkpoint's provenance.json records its source hashes and conversion
validation.
Validation
Before release, every checkpoint was checked for source identity, finite
tensors, shape compatibility, factor nonzero counts, and exact tensor equality
after the safetensors round-trip.
The release intentionally excludes base-model weights, activation Grams, dense
target/reconstructed matrices, optimizer state, data caches, remote-transfer
archives, credentials, and cluster-local paths.
The SWD release code and the checkpoints in this repository are distributed
under the Apache License 2.0. The GPT-2-derived checkpoints also retain the
upstream Modified MIT notice in
THIRD_PARTY_LICENSES/GPT2-MODIFIED-MIT.txt. See NOTICE for attribution.
