Parameter	Original	Tiny
num_hidden_layers	61	8
hidden_size	7168	1024
num_attention_heads	128	8
head_dim	512	64
q_lora_rank	1536	256
o_lora_rank	1024	64
o_groups	16	2
index_n_heads	64	4
index_head_dim	128	32
index_topk	1024	64
n_routed_experts	384	16
moe_intermediate_size	3072	512
sliding_window	128	64

Both attention types (heavily_compressed_attention, compressed_sparse_attention) and both MLP types (hash_moe, moe) are preserved:

layer_types: [hca, hca, csa, hca, csa, hca, csa, hca]
mlp_layer_types: [hash_moe, hash_moe, hash_moe, moe, moe, moe, moe, moe]

Checkpoint Structure

Single-file checkpoint (model.safetensors) with 608 tensors. The checkpoint uses the original DeepSeek-V4-Pro naming convention (no model. prefix): layers.N.attn.wq_a.weight, embed.weight, head.weight, etc. Fully loadable with AutoModelForCausalLM.from_pretrained().

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Pro-0.5B-A0.37B")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

Inspected DeepSeek-V4-Pro config to identify architecture: dual attention types (heavily_compressed_attention, compressed_sparse_attention), dual MLP types (hash_moe, moe), HyperConnection residual streams (hc_mult=4), and MTP layer.
Reduced all hidden dimensions while preserving at least one of each layer type.
Fine-tuned on copypasta dataset until perplexity < 3.0 (achieved 1.001).
Validated checkpoint structure matches original model naming conventions.

Notes

The model uses transformers naming convention for tensor keys (compatible with from_pretrained()).
A converted checkpoint matching the original DeepSeek-V4-Pro key naming is also available (model.safetensors with original layers.N.attn.wq_a.weight style names).
FP8 quantization scale tensors from the original model are not present in this unquantized bf16 checkpoint.
The num_nextn_predict_layers=1 (MTP) layer is preserved in the architecture.
Validation output: Success: 1.001 <= 10.0