Enrollment No.: 24B0908 Base model: Qwen/Qwen3-4B-Instruct-2507

One-shot, Hessian-aware unstructured weight pruning (Frantar & Alistarh, "SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot", ICML 2023), implemented directly from the reference source (https://github.com/IST-DASLab/sparsegpt) in sparsegpt_core.py. Each linear layer is pruned column-block by column-block using a calibration Hessian built from 128 C4 samples (H = 2 X^T X). Weights are scored by w^2 / [H^-1]_ii^2 and the removed weight's error is propagated to the weights still to be pruned via the inverse Hessian, so the surviving weights absorb the reconstruction error.

Pruned linears are stored as a packed 1-bit mask plus bf16 values for the surviving weights; everything else (embedding, norms) stays dense bf16. That puts pruned tensors at roughly 9 bits/weight (1 mask bit + 8 value bits), and the embedding table (~10% of parameters, never pruned) adds its full 16 bits on top, which is why the ratio lands at 60% rather than the nominal 50%.

70% sparsity gets the size down to ~42% of the original, closer to the 40% target, but the model at that sparsity collapses into repeating loops with no coherent answer — 0% on both benchmarks. 50% sparsity is the highest we tested that still produces a working model, so that's what this submission uses.

n=50 per benchmark. Full predictions in results/.

A real drop from baseline. The responses at this sparsity stay coherent and on-topic — the model just answers wrong more often. It never falls into the repetition loops that show up at 70%. During development we checked the pruned weights against the Hessian-based score to confirm the selection is real (it disagrees with plain magnitude pruning on about 10% of weights on a sampled layer), and confirmed the surviving weights are numerically adjusted by the error-compensation step. Details in sparsegpt_submission/DESIGN.md at the project root.

Unstructured sparsity doesn't speed up a standard dense GEMM kernel by itself — the packed mask shrinks the checkpoint on disk, but this submission decompresses to dense bf16 before running inference. The path to an actual CUDA speedup is NVIDIA's 2:4 semi-structured sparsity on Ampere and newer tensor cores (torch.sparse.SparseSemiStructuredTensor). SparseGPT supports this natively — sparsegpt_core.fasterprune already takes prunen=2, prunem=4 for exactly that pattern — this submission just doesn't use it, since a 2:4 mask is coarser than the calibration-driven unstructured mask used here and would cost some of the accuracy above.