Khazri 2 Mini is a compact, open-weight decoder-only language model in the Khazri family. It combines a modern LLaMA-style Transformer with a custom Byte-Level BPE tokenizer and a 2B-token English training corpus.

Hugging Face · Khazri · Contact

At a glance

Item	Detail
Model	Khazri 2 Mini
Parameters	100.68M
Status	Open weights on Hugging Face
Architecture	LLaMA-style, decoder-only Transformer
Training precision	bf16
Context configured for training	1,024 tokens
Vocabulary	32,768 tokens
Tokenizer	Custom Byte-Level BPE
Attention	Grouped-Query Attention: 12 query heads / 4 KV heads

Architecture

Component	Configuration
Hidden size	768
Transformer layers	12
Attention heads	12
Key/value heads	4
MLP intermediate size	2,048
Positional encoding	RoPE
Normalization	RMSNorm
MLP activation	SwiGLU / SiLU
Attention backend	FlashAttention-2 where available; PyTorch SDPA fallback
Embeddings	Tied input/output embeddings

Training data

Khazri 2 Mini is trained on a custom, pretokenized English-only corpus with a target size of 2,000,000,000 tokens. The corpus is packed into 1,953,125 sequences of 1,024 tokens and stored in Arrow shards with source identifiers.

The documented token budget is:

Source	Token budget	Share	Role
Cosmopedia	850M	42.5%	General English educational and synthetic-text coverage
The Stack v2 Dedup	450M	22.5%	Code from Python, JavaScript, TypeScript, Java, C++, C, Go and Rust
TinyStories	300M	15.0%	Simple narrative language
LIMA	50M	2.5%	Instruction and conversation examples
SYNAPSE synthetic instruction data	350M	17.5%	Arithmetic, context, abstention, web-needed, identity, symbolic-math and general-assistant routes

The listed values are the documented source-token budget. The release manifest should be used for the final source counts of a particular weight revision.

Data processing and safeguards

Only English text is retained for this corpus. Short records are excluded, very long records are capped, and language/character checks are applied before tokenization.
The code portion is limited to the eight languages listed above.
The training mix combines general English, narrative, code, instruction and route-aware synthetic material. This preserves general capabilities while teaching specialized SYNAPSE behaviours.
Checkpoint evaluation should cover general Q&A, code, arithmetic, context extraction, abstention and current-information requests. A route-specific gain should not be accepted if it degrades general behaviour.
Original source datasets remain subject to their own terms and licences. Consult their source pages and the model repository licence before use.

Tokenizer

Khazri 2 Mini uses a 32,768-token custom Byte-Level BPE tokenizer. It reserves structural whitespace, chat and SYNAPSE route tokens as single tokens, preserves indentation for code, and uses single-digit splitting to make arithmetic strings more explicit to the model.

Installation

pip install -U torch transformers accelerate safetensors

Quick start

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "Yusiko/khazri-2-mini"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "Write a concise explanation of a small language model."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Khazri 2 Preview comparison

The following table reports the project-provided compact-model comparison for Khazri 2 Preview, not Khazri 2 Mini. Higher is better for every listed task.

Model	Parameters	Context extraction	Mixed speed/proxy	Arithmetic	Word problems	Abstention
Khazri 2 Preview	~250M	100%	62%	99%	99%	97.4%
Gemma 3	270M	100%	36%	0%	0%	18%
Qwen 2.5	0.5B	89%	45%	14%	28%	46%
Pythia	160M	22%	10%	0%	2%	1%

These are internal preview results on selected compact-model tests. They are not independently audited and should not be used to make claims about Khazri 2 Mini. Publish prompts, model revisions, scoring rules, hardware and complete evaluation assets with any future benchmark announcement.

Responsible use

Khazri 2 Mini can produce incorrect, incomplete or biased outputs. Evaluate it on your own task, verify material claims and keep a human in the loop for consequential decisions. Do not rely on it as the sole basis for legal, medical, financial, hiring, safety or other high-impact decisions.

Roadmap

Khazri 2 Mini is part of the second Khazri generation. The next planned stage is Khazri 3: a larger parameter scale and stronger results.

Contact

For research, integration or partnership inquiries, visit khazri.dev or email contact@khazri.dev.