You can deploy this model and start earning money today!

Jiraya/
zoof-v1.2-394M-chat

Name: zoof-v1.2-394M-chat
Availability: InStock

instruction-following

sft

pytorch

wizardlm_evol_instruct_70k

Jiraya/zoof-v1.2-394M

finetune

mit

endpoints_compatible

Views

No views yet

Zoof-394M-Chat (v1.2.2)

The instruction-tuned variant of the zoof-v1.2-394M Small Language Model.

Model Summary

zoof-394M-chat is the instruction-tuned version of zoof-394M. It has been fine-tuned using Supervised Fine-Tuning (SFT) to follow instructions, answer questions, and engage in dialogue.

This model retains the efficiency of the base model while offering significantly improved coherence and adherence to user prompts.

💻 Usage

Please refer to the Zoof repository.

Training Data

Pre-Training: fineweb-edu
SFT: No Robots, WildChat-4.8M, Magpie Pro 300k Filtered and WizardLM Evol Instruct 70k.

📊 Performance & Benchmarks

The following table compares zoof-v1.2-394M (base model) against industry-standard baselines in the 300-500M parameter class.

Benchmark	Metric	zoof-394M (v1.2.2)	SmolLM-360M	SmolLM2-360M	Qwen2.5-0.5B
Training Tokens	Data Efficiency	79B	600B	4T	18T
PIQA	Physical Commonsense	69.5	71.6	71.7	69.9
BoolQ	Boolean Reasoning	59.9	-	-	-
WinoGrade	Pronoun Resolution	53.8	52.8	52.5	54.1
HellaSwag	Commonsense NLI	47.0	51.8	54.5	51.2
OBQA	OpenBookQA	37.2	37.2	37.4	37.4
ARC-E	Science (Easy)	44.3	-	-	-
ARC-C	Science (Challenge)	32.3	-	-	35.6
SIQA	Social Commonsense	40.3	-	-	-
MMLU cloze	General Knowledge	28.5	34.4	35.8	33.7
MMLU	General Knowledge	29.6	-	-	-
RACE	Reading Comprehension	38.3	-	-	-

Note: Zoof achieves these scores with ~2% of the training compute used for SmolLM2 (79B vs 4T tokens), highlighting the efficiency of the FineWeb-Edu dataset.

⚙️ Technical Specifications

Architecture: Decoder-only Transformer with RMSNorm (Pre-Norm).
Positional Embeddings: Rotary Positional Embeddings (RoPE).
Attention: Flash Attention via F.scaled_dot_product_attention.
Initialization: Depth-scaled initialization (1/\sqrt{2L}) for residual stability.
Context Length: 1024 Tokens.
Parameters: 394M.
Base Model: zoof-v1.2-394M
Language: English

Jiraya/zoof-v1.2-394M-chat