You can deploy this model and start earning money today!

ikimyaii/
hgrn-1.3B-obs-cancel-block-50pct

Name: hgrn-1.3B-obs-cancel-block-50pct
Availability: InStock

ikimyaii/hgrn-1.3B-dense-baseline

finetune

mit

Views

No views yet

HGRN-1.3B OBS-cancel-block 50% Sparse

HGRN-1.3B pruned to 50% sparsity using OBS-cancel-block, a post-training pruning method based on second-order weight correction with cancellation-aware selection. Used in:

When Does One-Shot Pruning Beat Iterative Optimisation? Second-Order Correction for Sparse LLMs on Neuromorphic Hardware
Kimia Gholami et al., NeurIPS 2026 submission

Performance

Metric	Value
WikiText-2 PPL (dense)	14.18
WikiText-2 PPL (sparse)	19.02
Sparsity	50%
ARC-Easy	0.461
ARC-Challenge	0.261
HellaSwag	0.427
PIQA	0.669
WinoGrande	0.525
LAMBADA	0.230
Average	0.429

This is the best-performing pruned model at this sparsity level, outperforming SparseGPT (PPL 20.3, avg 0.429), Wanda (PPL 404), RIA (PPL 410), and AWP (PPL 527).

Sparsity structure

Semi-structured sparsity: exactly k = floor(d_in * 0.5) weights are zeroed per output row in every linear layer. All 24 transformer blocks are pruned, including recurrent projections (i_proj, f_proj, g_proj, o_proj) and MLP layers (gate_proj, up_proj, down_proj), as well as lm_head.

This format maps directly to Intel Loihi: zero weights generate no spike events and cost no energy or memory bandwidth, making 50% sparsity a proportional reduction in active synaptic operations.

Method summary

OBS-cancel-block identifies cancellation groups: correlated input channels with mixed-sign weights whose joint removal cost is lower than the sum of their individual OBS saliency scores. Selection uses a greedy $r_j^2/d_j$ criterion with Schur complement updates, confined to non-overlapping blocks of 128 columns. After each selection step, exact OBS weight corrections are applied to surviving weights. Calibration used 64 batches (batch size 4, sequence length 512) from WikiText-2.

Loading

python

import torch
import fla
from fla.models.hgrn import HGRNConfig, HGRNForCausalLM
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

AutoConfig.register("hgrn", HGRNConfig, exist_ok=True)
AutoModelForCausalLM.register(HGRNConfig, HGRNForCausalLM, exist_ok=True)

model_id = "ikimyaii/hgrn-1.3B-obs-cancel-block-50pct"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
).cuda()

Install FLA first:

pip install flash-linear-attention