Parameter	Value
Architecture	LLaMA
Parameters	~50M
Hidden size	576
Layers	8
Attention heads	8
Vocab size	32,000
Context length	2,048 tokens
Training tokens	~0.51B (510M)
Precision	bf16

Training

Hyperparameter	Value
Learning rate	6e-4
Scheduler	Cosine
Warmup steps	500
Batch size	16 x 4 (grad accum) = 64
Epochs	3
Weight decay	0.1
Max grad norm	1.0

Results

Metric	Value
Train loss	3.38
Eval loss	3.379
Perplexity	29.35

Training Data

Corpus: Abzalbek89/corpus_clean — 1.5M cleaned Kazakh texts
Tokenized: Abzalbek89/corpus_clean_tokenized — 249K blocks x 2048 tokens
Tokenizer: Abzalbek89/kk-tokenizer-bpe-32k — ByteLevel BPE, 32K vocab

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Abzalbek89/kk-llama-50m")
tokenizer = AutoTokenizer.from_pretrained("Abzalbek89/kk-llama-50m")

input_text = "Қазақстан Республикасының"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

License

Apache 2.0