You can deploy this model and start earning money today!

Yusiko/
Khazri

Name: Khazri
Availability: InStock

Views

No views yet

🌪️Khazri — Azerbaijani Language Model

A lightweight, efficient, and fully custom Azerbaijani language model designed for text generation, chat applications, education, and research. Khazri is trained from scratch using a custom 10M-sample Azerbaijani dataset and optimized for running on consumer GPUs while maintaining competitive performance.

🌟 Features

🇦🇿 Native Azerbaijani language support
⚡ Lightweight architecture (≈ 36M parameter)
🚀 Supports fast inference with GGUF + llama.cpp
📦 Available on Hugging Face
🎯 Optimized for chatbots, WebRTC real-time assistants, and low-latency deployment

🏗️ Model Architecture

Version	Parameters	Type	Context Length	Notes
Khazri-36M	~36.6M	GPT-2 Small variant	1024	Higher quality

Architecture:

Transformer decoder-only
Multi-head self-attention
Rotary positional embeddings (RoPE)
GELU activation
Layer normalization
Tied embeddings

📚 Dataset

Khazri is trained on a 10 million-sample Azerbaijani dataset including:

News, books, conversations, social media, web articles, educational content

Preprocessing:

Unicode normalization, deduplication, tokenizer preprocessing, length filtering

🏋️ Training Details

Hardware

NVIDIA RTX 3090 24GB
PyTorch 2.x + CUDA 12
bf16 mixed precision

Hyperparameters

epochs = 1
batch_size = 32
gradient_accumulation = 4
learning_rate = 3e-4
warmup_steps = 500
weight_decay = 0.1
sequence_length = 512
optimizer = AdamW
precision = bf16

📈 Training Challenges & Solutions

Bottleneck: Memory Bandwidth

Small models saturate VRAM bandwidth → ~4.2 it/s
Solution: shrink model size, adjust batch/accumulation, optimize data loading

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Yusiko/Khazri")
model = AutoModelForCausalLM.from_pretrained("Yusiko/Khazri")

🌐 Hugging Face

Available at: https://huggingface.co/Yusiko/Khazri

📦 License

GPL 3.0 License

🌍 Future Plans

1B+ model
Better tokenizer
Instruction-tuning
WebGPU inference
Community fine-tuning tools

🤝 Contact

Created by Yusiko
GitHub: Yusiko99
Website: https://yusi.xo.je
Hugging Face: Yusiko