This repository contains a tiny DeepseekV3ForCausalLM Mixture-of-Experts language model trained from scratch on the full TinyStories training corpus.

The model has 2,803,272 total parameters and approximately 1,807,944 parameters active per token. It retains the main DeepSeek-V3 inference building blocks at a deliberately small scale: Multi-Head Latent Attention (MLA), Q and KV low-rank compression, interleaved RoPE, routed and shared experts, sigmoid top-k routing, and correction-bias-based load balancing.

This is an independently trained synthetic tiny checkpoint. It is not an official DeepSeek model, contains no weights from an original DeepSeek checkpoint, and should not be expected to match the capabilities of production DeepSeek models.

Repository contents

hf/: the final Hugging Face checkpoint and tokenizer
example_generate.py: a minimal local generation example
eval_text_generation.json: generations from the final training evaluation
artifact_metadata.json: training arguments, metrics, router usage, and the expanded configuration
deepseek_v3_config_dump.json: a standalone configuration dump

Optimizer checkpoints, packed training data, and the full training log are intentionally omitted from the distribution package.

Architecture identity

This checkpoint loads through the standard Transformers classes:

text

DeepseekV3Config
DeepseekV3ForCausalLM
model_type: deepseek_v3

It does not use DeepseekV2ForCausalLM with a renamed model card. Important V3-specific behavior includes sigmoid routing, normalized selected routing weights, interleaved RoPE, Q LoRA compression, and an expert correction bias.

Model architecture

yaml

architecture: DeepseekV3ForCausalLM
model_type: deepseek_v3
total_parameter_count: 2,803,272
active_parameters_per_token: 1,807,944

model_vocab_size: 1,024
tokenizer_size: 1,003
hidden_size: 216
intermediate_size: 432
num_hidden_layers: 5
first_dense_layers: 1
moe_layers: 4

num_attention_heads: 8
num_key_value_heads: 8
q_lora_rank: 64
kv_lora_rank: 64
qk_nope_head_dim: 16
qk_rope_head_dim: 16
qk_head_dim: 32
v_head_dim: 32
rope_interleave: true
rope_theta: 10,000

routed_experts_per_moe_layer: 4
shared_experts_per_moe_layer: 1
experts_selected_per_token: 1
moe_intermediate_size: 128
n_group: 1
topk_group: 1
norm_topk_prob: true
routed_scaling_factor: 2.5

tie_word_embeddings: true
rms_norm_eps: 1.0e-6
max_position_embeddings: 2,048

Multi-Head Latent Attention

Every layer uses DeepSeek MLA rather than conventional GQA. Queries are compressed through a rank-64 Q projection, while keys and values share a rank-64 latent representation before expansion to eight attention heads. Each query/key head combines a 16-dimensional non-positional component and a 16-dimensional rotary component. Values use 32 dimensions per head.

num_key_value_heads equals num_attention_heads because MLA compresses the shared KV latent state before expanding it to the attention heads. Setting a smaller KV-head count would describe GQA, not this MLA implementation.

DeepSeekMoE routing

The first decoder layer is dense. Each of the remaining four layers contains four routed experts and one always-active shared expert. One routed expert is selected for each token.

The V3 router computes sigmoid affinity scores. Selection uses the affinity score plus a non-gradient correction bias, while the routed expert weight uses the original affinity score. During training, the correction bias was adjusted after every batch: overloaded experts were decreased and underloaded experts were increased. A very small complementary sequence-wise balance objective was also used.

yaml

v3_bias_update_speed: 0.01
sequence_aux_loss_alpha: 0.0001
no_token_dropping: true

All routed experts received traffic. Aggregate fractions across the complete training run ranged from approximately 0.140 to 0.496 depending on layer and expert. These aggregate values reflect learned routing specialization and are not expected to be exactly uniform.

MTP scope

The checkpoint contains the standard Transformers DeepSeek-V3 causal language model used for inference. It does not contain a training-only Multi-Token Prediction (MTP) draft module:

yaml

num_nextn_predict_layers: 0
training_objective: next-token prediction

Transformers 5.14.1 exposes the V3 MTP count in configuration but does not instantiate or train the original DeepSeek-V3 MTP module in DeepseekV3ForCausalLM. The checkpoint therefore preserves the V3 main-model inference graph and V3 routing behavior, but does not claim to reproduce the complete original V3 pretraining recipe.

Training data

The model was trained on the full TinyStories training corpus using an independent 1% validation split:

yaml

selected_stories: 2,119,489
training_stories: 2,098,294
validation_stories: 21,195
validation_fraction: 0.01

training_blocks: 2,441,053
validation_blocks: 24,607
block_size: 256

Stories were joined into a continuous packed stream:

BOS + story 1 + EOS + BOS + story 2 + EOS + ...

The stream was split into fixed 256-token blocks without per-story padding. Only the final incomplete block was discarded.

Tokenizer

The checkpoint uses a custom byte-level BPE tokenizer. The base vocabulary was trained on the 5% TinyStories experiment and frozen for the full run:

text

BPE()
ByteLevel(add_prefix_space=False)
base_vocab_size: 1,000
min_frequency: 2
normalizer: None

Special tokens use fixed IDs:

text

<s>          -> 1000
</s>         -> 1001
<|im_start|> -> 1002

<s> is both BOS and padding, while </s> is EOS. The model reserves 1,024 embedding rows and the tokenizer exposes 1,003 tokens.

The presence of <|im_start|> does not make this a chat model. The checkpoint is pretrained only and has no chat template or instruction tuning.

Training setup

The checkpoint was trained from scratch in float32 on an NVIDIA GeForce RTX 5060 Ti:

yaml

dtype: float32
batch_size: 16
block_size: 256
training_steps: 152,565
epochs: 1.0
tokens_processed: 624,906,240

optimizer: AdamW
learning_rate: 2.0e-4
warmup_steps: 2,000
scheduler: warmup + cosine decay
minimum_learning_rate: 2.0e-5
weight_decay: 0.0
grad_clip: 1.0

optimizer_loop_time: approximately 1h 03m 36s

Evaluation

The final checkpoint produced:

yaml

final_train_loss: 1.5201
validation_loss: 1.5254
validation_perplexity: 4.5972

Validation loss was computed over 32 batches, or 131,072 tokens, from the independent packed validation split. These values are compact checkpoint diagnostics, not general language-model benchmark results.

Example generation

A representative final sampled generation begins:

text

Once upon a time, there was a little girl named Lily. She had long black,
white hair that she loved to play with every day. One day, Lily's mom asked
her to clean up her bedroom. Lily didn't want to clean up, so she started to
pick up her toys and put them away in the closet.

An independent greedy reload test produced:

text

Once upon a time, there was a little girl named Lily. She loved to play
outside in the sunshine and pick flowers. One day, she saw a big, scary dog
running towards her. The dog ran away and Lily was sad.

Sampling is stochastic. The model produces recognizable TinyStories-style English, but contradictions, incorrect pronouns, invented words, repetition, and unfinished stories remain possible at this size.

Usage

Install the requirements:

pip install -r requirements.txt

Run the included example from the repository root:

python example_generate.py

Local loading:

python

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shibatch/tinydeepseekv3-3m"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_dir = Path(snapshot_download(
    repo_id=repo,
    allow_patterns=["hf/*"],
)) / "hf"

tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    dtype=torch.float32,
).to(device).eval()

input_ids = torch.tensor(
    [[tokenizer.bos_token_id] + tokenizer.encode(
        "Once upon",
        add_special_tokens=False,
    )],
    dtype=torch.long,
    device=device,
)

with torch.inference_mode():
    output = model.generate(
        input_ids=input_ids,
        max_new_tokens=100,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
        top_k=40,
        repetition_penalty=1.1,
        use_cache=True,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Loading the hf subdirectory from Hugging Face Hub:

python

from pathlib import Path
from huggingface_hub import snapshot_download

repo_dir = Path(snapshot_download(
    repo_id="shibatch/tinydeepseekv3-3m",
    allow_patterns=["hf/*"],
))
model_dir = repo_dir / "hf"

Use model_dir with the local loading example above.

Loading requirements

The model was trained, saved, and independently reloaded with:

text

transformers 5.14.1
torch 2.12.0+cu130

It requires a Transformers release containing DeepseekV3ForCausalLM. No custom model code or trust_remote_code=True is required.

Intended uses

This model is intended for:

testing DeepseekV3Config and DeepseekV3ForCausalLM
testing Multi-Head Latent Attention and compressed KV caching
testing Q and KV low-rank projections
testing V3 sigmoid MoE routing and shared experts
testing correction-bias-based expert load balancing
compact inference-engine and architecture experiments
testing generate(), save_pretrained(), and from_pretrained()

It is not intended for:

instruction following or chat
factual question answering
reasoning benchmarks
production deployment
safety-critical use
comparison with full-size DeepSeek models

Limitations

only 2.80 million total parameters
small 1,003-token tokenizer
English TinyStories-only pretraining
no instruction tuning and no chat template
no MTP training module or MTP objective
weak factual knowledge and reasoning
occasional grammatical and semantic errors
possible mojibake inherited from training-text byte sequences
no capability-equivalence claim with official DeepSeek models

Citation and references

This is a synthetic tiny DeepSeek-V3-compatible MoE checkpoint trained from scratch on TinyStories. It is intended for implementation validation, debugging, education, and small-scale architecture experiments.