Supervised fine-tune of deepseek-ai/DeepSeek-V2-Lite (15.7B total / 2.4B active params, 64 experts top-6 + 2 shared, MLA attention) on nvidia/OpenMathInstruct-2 train_1M.

Training setup

Stack: torchtitan 0.2.2 + DeepEP + NVSHMEM IBGDA + PyTorch 2.11
Hardware: 2 × 8 × NVIDIA H200 (16 GPU)
Parallelism: FSDP across 16 ranks + Expert Parallel (EP=16) cross-node, expert dispatch via DeepEP/IBGDA
Precision: bfloat16, AdamW, loss-only torch.compile
Hyperparameters:
- Learning rate: 2e-5, cosine decay, 5% warmup (93 steps)
- Effective batch size: 128 (16 GPU × 8 local × 1 grad-accum)
- Sequence length: 2048 packed
- Steps: 1864 (one epoch over train_1M post-packing)
- Wall-clock: ~44 minutes
Steady-state MFU: ~20% at EP=16 cross-node + DeepEP/IBGDA

Prompt template

The model was fine-tuned with the following template, with loss masking applied to the prompt portion (only the response contributes to the SFT loss):

Solve the following math problem step by step:

{problem}

Solution:
{response}<|end_of_sentence|>

For best results at inference time, use the same template structure.

Loss curve

Step	Loss
1	0.650
150	0.387
750	0.320
1500	0.313
1864 (final)	~0.32

Limitations

Trained for math-instruction-following specifically; not a general chat model.
Inherits any limitations of the DeepSeek-V2-Lite base.
No RLHF or preference tuning beyond the SFT phase.

aandraca/openmath-deepseek-v2-lite

DeepSeek-V2-Lite-OpenMath-SFT

Training setup

Prompt template

Loss curve

Limitations

aandraca/
openmath-deepseek-v2-lite