The base checkpoint is a tiny architecture-development version of Kimi K3. Its weights were initialized from scratch and trained only on a toy copypasta dataset. It is not a distilled or compressed version of the trained 2.8T Kimi K3 model.

This checkpoint is therefore an educational research artifact, not a production language model.

Training

Dataset: kz-transformers/multidomain-kazakh-dataset
Objective: causal language modeling
Optimizer steps: 200
Tokens processed: 1,638,400
Sequence length: 512
Effective batch: 8,192 tokens
Initial learning rate: 3e-4
Hardware: 1× NVIDIA T4
Runtime: approximately 12.7 minutes
Tokenization: native Kimi tokenizer accelerated with Gigatoken

Results

Metric	Before	After
Validation loss	16.7615	4.9539
Validation perplexity	19,030,254	141.72

Example generations

The model has started learning Kazakh Cyrillic, morphology, punctuation, and sentence structure, but its generations remain semantically incoherent and contain invented words. This is expected after only 1.64M training tokens.

Usage

python

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "Eraly-ml/Kimi-K3-0.40B-Kazakh-CPT-step200"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModel.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.float16,
    device_map="auto",
)

inputs = tokenizer(
    "Қазақстан —",
    return_tensors="pt",
).to(model.device)

output = model.language_model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))

Intended use

This checkpoint is intended for:

studying Kazakh language adaptation; inspecting Kimi K3's tiny MoE architecture; testing dataset and pretraining pipelines; comparing intermediate language acquisition checkpoints.

It should not be used for factual, safety-critical, or production tasks.