You can deploy this model and start earning money today!

Jiraya/
zoof-v1.2-394M

Name: zoof-v1.2-394M
Availability: InStock

custom-implementation

HuggingFaceFW/fineweb-edu

mit

endpoints_compatible

Views

No views yet

Zoof-394M (v1.2.2)

A clean, optimized, and interpretable implementation of a decoder-only Transformer in PyTorch.

Model Summary

zoof-394M (v1.2.2) is a general-purpose language model pre-trained on 79 Billion tokens of high-quality educational data. It demonstrates how modern architectural choices and high-quality data can yield competitive performance in the sub-400M parameter regime, even with limited compute.

Despite its compact size, zoof-394M achieves competitive zero-shot performance on reasoning and commonsense benchmarks, often rivaling models trained on significantly larger datasets.

💻 Usage

Please refer to the Zoof repository.

Training Data

Pre-Training: fineweb-edu

📊 Performance & Benchmarks

The following table compares zoof-v1.2-394M (in zero shot setting) against industry-standard baselines in the 300-500M parameter class.

Benchmark	Metric	zoof-394M (v1.2.2)	SmolLM-360M	SmolLM2-360M	Qwen2.5-0.5B
Training Tokens	Data Efficiency	79B	600B	4T	18T
PIQA	Physical Commonsense	69.5	71.6	71.7	69.9
BoolQ	Boolean Reasoning	59.9	-	-	-
WinoGrade	Pronoun Resolution	53.8	52.8	52.5	54.1
HellaSwag	Commonsense NLI	47.0	51.8	54.5	51.2
OBQA	OpenBookQA	37.2	37.2	37.4	37.4
ARC-E	Science (Easy)	44.3	-	-	-
ARC-C	Science (Challenge)	32.3	-	-	35.6
SIQA	Social Commonsense	40.3	-	-	-
MMLU cloze	General Knowledge	28.5	34.4	35.8	33.7
MMLU	General Knowledge	29.6	-	-	-
RACE	Reading Comprehension	38.3	-	-	-

Note: Zoof achieves these scores with ~2% of the training compute used for SmolLM2 (79B vs 4T tokens), highlighting the efficiency of the FineWeb-Edu dataset.

⚙️ Technical Specifications

Architecture: Decoder-only Transformer with RMSNorm (Pre-Norm).
Positional Embeddings: Rotary Positional Embeddings (RoPE).
Attention: Flash Attention via F.scaled_dot_product_attention.
Initialization: Depth-scaled initialization ($1/\sqrt{2L}$) for residual stability.
Context Length: 1024 Tokens.
Parameters: 394M.
Chat Model: Zoof-v1.2-394M-chat
Language: English

Jiraya/zoof-v1.2-394M