You can deploy this model and start earning money today!

mohammad-mozaffari/
llama_3.2_1b-PATCH-45Sparse

Name: llama_3.2_1b-PATCH-45Sparse
Availability: InStock

meta-llama/Llama-3.2-1B

finetune

llama3.2

endpoints_compatible

Views

No views yet

PATCH

llama_3.2_1b-PATCH-45Sparse

This checkpoint (LLaMA-3.2 1B, PATCH-Joint, 45% sparsity): 42.08% average zero-shot accuracy, 12.23 WikiText2 perplexity.

Paper

GitHub

This repository hosts a mask only release for the paper PATCH: Learnable Tile-level Hybrid Sparsity for LLMs. PATCH (Pruning with a Learnable Tile-level Configuration for Hybrid Sparsity) learns a structured mask on frozen pretrained weights, assigning each tile as dense (0% sparsity) or 2:4 sparse (50% sparsity) to hit a flexible global sparsity target while staying hardware-friendly.

Because PATCH/MaskLLM keep the base weights frozen, we distribute only the binary keep/prune mask (bit-packed in mask.npz) - no weight values. You recover the sparse model by downloading the original base model and applying the mask.

Base model: meta-llama/Llama-3.2-1B
Method: PATCH-Joint | Target sparsity: 45% | Pattern: Dense / 2:4 tiles
Measured mask sparsity: 44.99%.

Results (LLaMA-3.2 1B)

Sparsity	Method	Pattern	Avg Acc (% ↑)	WikiText2 PPL (↓)
0%	Dense	-	47.70	9.06
50%	Magnitude	2:4	29.66	563.44
50%	Wanda	2:4	31.61	78.18
50%	SparseGPT	2:4	35.55	32.73
50%	Thanos	2:4	35.71	33.03
50%	ProxSparse	2:4	33.55	49.33
50%	MaskLLM	2:4	41.04	12.93
45%	PATCH-Joint ⭐	Dense/2:4	42.08	12.23
35%	PATCH-Joint	Dense/2:4	42.72	11.67
25%	PATCH-Joint	Dense/2:4	43.81	11.00

Per-task zero-shot accuracy (%) for this checkpoint:

MMLU	PIQA	ARC-E	ARC-C	WinoG.	OBQA	RACE	HellaS.	Average
23.81	70.89	60.77	27.22	56.27	22.80	34.07	40.78	42.08

All numbers are from the PATCH paper (arXiv:2509.23410); accuracy is the average over MMLU, PIQA, ARC-Easy, ARC-Challenge, Winogrande, OpenBookQA, RACE and HellaSwag, evaluated with the LM-Evaluation-Harness. PPL is WikiText2.

Training hyper-parameters

Hyper-parameter	Value
Fine-tuning dataset	SlimPajama (2B tokens)
Training steps	2000
Global batch size	256
Sequence length	4096
Mask tile size	128 x 128 (hardware tiles: 128x128 / 128x64 / 64x128 / 64x64)
Logits init.	N(0, 0.014)
Tile-logit prior	SparseGPT (strength 3)
Regularization scope	Global (single target density)
Evaluation	LM-Eval-Harness (8 zero-shot tasks) + WikiText2 PPL @ seqlen 4096
Hardware	1 node x 4 GPUs, data parallel (HuggingFace Trainer)
Optimizer	Adam
Learning rate	1e-3
Gumbel scaling (kappa)	25 -> 350
Gumbel temp (tau)	4 -> 0.05
Sparsity reg. (lambda1)	7
Weight reg. (lambda2)	10

How to use

python

from huggingface_hub import hf_hub_download
from transformers import AutoModelForCausalLM
import torch
from load_patch_mask import apply_patch_mask  # shipped in this repo

npz = hf_hub_download(repo_id="mohammad-mozaffari/llama_3.2_1b-PATCH-45Sparse", filename="mask.npz")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B", torch_dtype=torch.bfloat16)
apply_patch_mask(model, npz)   # zeroes the pruned weights in place

Or from the command line:

python load_patch_mask.py --base_model meta-llama/Llama-3.2-1B --mask_repo mohammad-mozaffari/llama_3.2_1b-PATCH-45Sparse

Speedup on real hardware requires a 2:4-aware / hybrid sparse kernel; see the GitHub repository and STOICC.

License

The released mask is a derivative of the base model and is distributed under the base model's license (llama3.2). You must comply with that license and obtain access to the base model separately.

Built with Llama. Use governed by the Llama 3.2 Community License.

The mask-generation code is released under the MIT license (see the PATCH repository).

Citation

bibtex

@article{hourri2025patch,
    title  = {PATCH: Learnable Tile-level Hybrid Sparsity for LLMs},
    author = {Hourri, Younes and Mozaffari, Mohammad and Mehri Dehnavi, Maryam},
    year   = 2025,
    journal = {arXiv preprint arXiv:2509.23410}
}