You can deploy this model and start earning money today!

mkd-hossain/
keural-stage1-checkpoints

Name: keural-stage1-checkpoints
Availability: InStock

Views

No views yet

Keural 14.8B — Stage 1 Pretraining Checkpoints

This repository contains raw pretraining checkpoints from Stage 1 of the Keural 14.8B MoE language model, trained from scratch on bilingual Korean-English data.

These are not instruction-tuned models. They are raw base checkpoints useful for research, resuming training, or fine-tuning experiments.

Model Overview

Field	Value
Architecture	Mixtral-style MoE (Mixture of Experts)
Parameters	14.83B total (2.6B active per token)
Layers	24 transformer blocks
Hidden size	4096
Attention heads	32 (8 KV heads, GQA)
Experts	8 total, top-2 routing
Vocab size	131,072 (custom SentencePiece)
Context length	4096 tokens
Languages	Korean + English
Training hardware	2× NVIDIA H200 150GB, FSDP
Infrastructure	KT Cloud NIPA2-H200

Training Data

Stage 1 pretraining used ~64.56B tokens of bilingual web text:

Source	Language	Tokens
FineWeb	English	~40B
WanJuan	Chinese/English	~10B
HPLT	Korean	~8B
FineWeb2	English	~6B

Data was filtered for quality (spam removal, length filtering, deduplication) and tokenized with the custom Keural SentencePiece tokenizer (vocab=131,072).

Training Configuration

Parameter	Value
Total steps	100,000
Batch size	~645 tokens × large batch
Total tokens	~64.56B
Optimizer	AdamW
Learning rate	Cosine schedule with warmup
Precision	bfloat16
Parallelism	FSDP (2× H200)

Checkpoints in This Repository

File	Step	Tokens Seen	Notes
`checkpoint_19000.pt`	19,000	~12.3B	Early training
`checkpoint_25000.pt`	25,000	~16.1B	Early training
`checkpoint_25500.pt`	25,500	~16.5B	Early training
`checkpoint_36000.pt`	36,000	~23.2B	Mid training
`checkpoint_50000.pt`	50,000	~32.3B	Mid training
`checkpoint_65000.pt`	65,000	~41.9B	Late training
`checkpoint_70000.pt`	70,000	~45.2B	Late training
`checkpoint_75000.pt`	75,000	~48.4B	Late training
`checkpoint_80000.pt`	80,000	~51.6B	Late training
`checkpoint_85000.pt`	85,000	~54.9B	Late training
`checkpoint_90000.pt`	90,000	~58.1B	Late training
`checkpoint_95000.pt`	95,000	~61.3B	Late training
`checkpoint_100000.pt`	100,000	~64.56B	Stage 1 Final

Each .pt file is a full FSDP training checkpoint (~83GB) containing:

model — model state dict
optimizer — optimizer state
step — training step number
loss — training loss at that step

Full Training Pipeline

This is Stage 1 of the full Keural training pipeline:

Stage 1: Pretraining     → 100K steps, ~64.56B tokens  ✅ Done (this repo)
Stage 2: Annealing       → 20K steps,  ~5.16B tokens   ✅ Done
Stage 3: SFT             → 18K steps,  1.13M samples   🔄 In Progress
Stage 4: DPO             → ~8K steps,  233K pairs       ⏳ Planned

For the final instruction-tuned model, see: mkd-hossain/keural-14.8b-sft

Loading a Checkpoint

These checkpoints use the custom Keural model architecture. You need the training code to load them:

python

import torch

ckpt = torch.load("checkpoint_100000.pt", map_location="cpu", weights_only=False)
state_dict = ckpt["model"]
step = ckpt["step"]        # 100000
loss = ckpt["loss"]        # training loss

print(f"Step: {step} | Loss: {loss:.4f}")
print(f"Keys: {list(state_dict.keys())[:5]}")

For inference, use the vLLM-converted safetensors version: 👉 mkd-hossain/keural-14.8b-stage2-vllm

Tokenizer

The model uses a custom SentencePiece tokenizer with 131,072 vocab size, trained on Korean and English text:

python

import sentencepiece as spm

sp = spm.SentencePieceProcessor()
sp.Load("keural_tokenizer.model")

ids = sp.Encode("안녕하세요, Hello!", out_type=int)
text = sp.Decode(ids)

Tokenizer available at: mkd-hossain/keural-14.8b-base

Intended Use

Research on bilingual Korean-English language models
Fine-tuning experiments starting from a pretrained base
Studying MoE model behavior at different training stages
Resuming or extending pretraining

Not intended for: Direct deployment without instruction tuning. These are raw pretrained checkpoints.

Citation

bibtex

@misc{keural-14.8b-2026,
  title  = {Keural 14.8B: A Bilingual Korean-English MoE Language Model Trained from Scratch},
  author = {MKD Hossain},
  year   = {2026},
  url    = {https://huggingface.co/mkd-hossain/keural-stage1-checkpoints}
}

Keural 14.8B — Trained from scratch on KT Cloud NIPA2-H200 infrastructure. MKD-CORP | 2026