You can deploy this model and start earning money today!

tjingrant/
sparsellm-1b-20p-small-dense

Name: sparsellm-1b-20p-small-dense
Availability: InStock

Views

No views yet

tjingrant/sparsellm-1b-20p-small-dense

This is a small dense language model serving as a quality reference for sparsellm-1b model with 20% sparsity.

Model Details

Model Type: Sparse Causal Language Model
Base Model: LLaMA2-1B
Sparsity Configuration: Small-Dense architecture with 20%-20% density pattern
Training Data: Trained on the Fineweb-Edu dataset
Tokenizer: Same as the original LLaMA2 model
Perplexity: 19.49 (measured on Wikitext-103)
Parameter Counts:
- Total Parameters: 1.07B
- Total Linear Parameters: 1.01B
- Non-zero Linear Parameters: 1.01B
- Linear Layer Sparsity: 0.00%
- Average Linear Parameters During Training: 1.01B

Training Parameters

Training Steps: 11200
Batch Size: 8M tokens (4096 × 2048)
Learning Rate: 0.0003
Total Training Tokens: 89600000000 (89.6B)
Final Training Loss: 2.1385 ± 0.0224 (from last 1% of steps)
Pruning Start Step: 2500
Pruning End Step: 8100
Matching Sparse Model: sparsellm-1b-20p

Performance and Training Details

Here is the performance and parameter information for all models in this series:

Model	Total Params	Linear Params	Avg Linear Params	Non-Zero Linear	Sparsity	Batch Size	LR	Total Tokens	Final Train Loss	Perplexity
sparsellm-1b-20p	1.20B	1.14B	1.02B	0.91B	20.00%	8M	3e-4	89.6B	2.133 ± 0.022	19.58
sparsellm-1b-40p	1.20B	1.14B	0.87B	0.68B	40.00%	8M	3e-4	104.4B	2.137 ± 0.013	19.93
sparsellm-1b-60p	1.20B	1.14B	0.69B	0.46B	60.00%	8M	3e-4	131.0B	2.182 ± 0.017	20.80
sparsellm-1b-80p	1.20B	1.14B	0.45B	0.23B	80.00%	8M	3e-4	200.4B	2.228 ± 0.021	25.77
sparsellm-1b-20p-small-dense	1.07B	1.01B	1.01B	1.01B	0.00%	8M	3e-4	89.6B	2.139 ± 0.022	19.49
sparsellm-1b-40p-small-dense	0.88B	0.82B	0.82B	0.82B	0.00%	8M	3e-4	104.4B	2.161 ± 0.024	21.40
sparsellm-1b-60p-small-dense	0.70B	0.65B	0.65B	0.65B	0.00%	8M	3e-4	131.0B	2.209 ± 0.021	22.58
sparsellm-1b-80p-small-dense	0.46B	0.42B	0.42B	0.42B	0.00%	8M	3e-4	200.4B	2.237 ± 0.028	24.57

Notes:

Perplexity is measured on Wikitext-103
Batch Size is given in tokens (samples × sequence length)
Total Tokens = Training Steps × Batch Size
Final Train Loss is computed from the last 1% of training steps (mean ± std)
Avg Linear Params is the average number of active parameters during training, computed from the pruning schedule
Rows 1-4 are sparse models, rows 5-8 are the matching dense models with (approximately) matching average parameter counts over pretraining.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("{model_name}")
model = AutoModelForCausalLM.from_pretrained("{model_name}")

inputs = tokenizer("Hello, my name is", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model in your research, please cite our paper:

The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws

bibtex

@inproceedings{{
jin2025the,
title={{The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws}},
author={{Tian Jin and Ahmed Imtiaz Humayun and Utku Evci and Suvinay Subramanian and Amir Yazdanbakhsh and Dan Alistarh and Gintare Karolina Dziugaite}},
booktitle={{The Thirteenth International Conference on Learning Representations}},
year={{2025}},
}}