You can deploy this model and start earning money today!

SurjoLabs/
Blaze

Views

No views yet

Blaze (48.3M)

Blaze is a 48.3M parameter causal language model developed by SurjoLabs. It scores 15.45 on the Intelligence Index, placing #1 in the sub-50M parameter category on the Open SLM Leaderboard.

The model uses XSA (orthogonal value-subtraction) attention with recurrent layer sharing, achieving an effective computational depth of 26 layers while storing only 14 physical layers.

Architecture Specifications

Parameter	Value
Total Parameters	48,251,136
Physical Layers	14 (1 prelude + 12 recurrent + 1 coda)
Recurrent Passes	2 (effective depth: 26 layers)
Hidden Size	512
Intermediate Size	1536
Attention Heads	8 Query, 4 Key-Value (2:1 GQA)
Head Dimension	64
Vocabulary Size	8,192 (tied embeddings)
Context Length	1,024 tokens

Training & Checkpoint Selection

Total Tokens: ~20.97B tokens (20,000 steps at 2^20 = 1,048,576 tokens/step)
Schedule: WSD (Warmup-Stable-Decay) learning rate scheduler
Selected Checkpoint: Checkpoint 19,500 achieved peak performance across benchmarks and is the official set of weights released in this repository.

Benchmark Results

Evaluated 0-shot using normalized accuracy (acc_norm):

Benchmark	Score
PIQA	62.51%
ARC-Easy	41.84%
ArithMark-3.0	37.80%
HellaSwag	31.84%
ARC-Challenge	24.91%
Intelligence Index	15.45

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SurjoLabs/Blaze"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).cuda()

prompt = "The speed of light is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=32)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

MIT