DeepSeek-V4-Pro-Qwen3.5-4B is a reasoning-focused fine-tune of Qwen3.5-4B, distilled from responses generated by DeepSeek-V4-Pro in Max Effect mode. It follows the same training pipeline and approximately 250,000-sample mathematics and STEM mixture used for the 9B release, while targeting a substantially smaller and more accessible deployment class.
🧩 Structured ReasoningLearns multi-step solution patterns and explicit problem decomposition from a strong teacher.
📐 Math & STEM FocusTrained on approximately 250,000 samples centered on mathematical and scientific reasoning.
⚡ Compact DeploymentBrings the DeepSeek-distilled training recipe into the efficient 4B parameter class.
🧪 Reproducible EvaluationEvaluated on complete GSM8K runs and fixed MMLU-Pro Math, Physics, and Chemistry samples.
🤝 Collaboration & Training Support
🧪 Hardware Cooperation & Joint Collaboration
This project is built in close collaboration with hardware engineer Kyle Hessling, whose infrastructure, training support, and evaluation assistance helped make this release possible.
👉 Follow his hardware and model-training updates on X / Twitter: @KyleHessling1
🦥 Fine-tuning Framework (Unsloth)
The model training workflow is accelerated and memory-optimized with Unsloth. Special thanks to the Unsloth team for making efficient large-model fine-tuning more accessible.
🧬 The 9B Training Recipe, Compressed into a 4B Student
The teacher data for this release was generated with DeepSeek-V4-Pro (Max Effect). The 4B student uses the same training flow and approximately 250,000 mathematics and STEM samples as DeepSeek-V4-Pro-Qwen3.5-9B, emphasizing structured derivation, problem decomposition, verification, and reliable final-answer construction.
Training scope: no coding data was included in this SFT mixture. The published benchmarks in this card therefore focus on mathematics and STEM; coding and tool-use transfer have not yet been established for the 4B release.
Why coding coverage is currently limited: during this training cycle, DeepSeek released newer 0731 and 0813 models with substantially improved coding and agentic capabilities. The corresponding new teacher data was not yet available when this distillation run was prepared. Once the new model data is collected and prepared, this distillation line will be iteratively upgraded with dedicated coding and agent traces.
1 · Teacher Generation DeepSeek-V4-Pro in Max Effect mode produces detailed mathematics and STEM solutions.
2 · Data Preparation Approximately 250K samples are cleaned and formatted for supervised reasoning transfer.
3 · Unsloth SFT Qwen3.5-4B is fine-tuned through an Unsloth LoRA pipeline and merged into BF16.
4 · Deployment Build An MTP-enabled Q8 GGUF build is used for the reported llama.cpp evaluations.
📊 2. Benchmark Results
⚡ Performance Snapshot
Strong grade-school mathematics performance from a compact 4B model, with completed 1,500-question MMLU-Pro STEM results.
GSM8K91.77%2-run pass@1 mean
MMLU-Pro Math80.40%402 / 500
MMLU-Pro Physics74.20%371 / 500
MMLU-Pro Chemistry74.80%374 / 500
MMLU-Pro Average76.47%1,147 / 1,500
2.1 GSM8K
The MTP-Q8 build was evaluated twice on the complete 1,319-question GSM8K test split. Across 2,638 sampled answers, the model produced 2,421 exact-answer matches, yielding a two-run pass@1 mean of 91.7741%. The two independent run scores were 91.6603% and 91.8878%, a spread of only 0.23 percentage points.
Model
Accuracy
MiMo-V2.5-Pro
99.60%
Llama-3.1-405B-Instruct
96.80%
Llama-3.3-70B-Instruct
94.84%
DeepSeek-V4-Pro-Qwen3.5-9B
94.50%
DeepSeek-V4-Pro
92.60%
DeepSeek-V4-Pro-Qwen3.5-4B
91.77%
DeepSeek-V3
89.30%
📈 Compact-model result The 4B student is 2.73 points behind the published 9B score while remaining 2.47 points above the DeepSeek-V3 reference shown in the 9B release card.
Comparison note: the 4B result is a two-run MTP-Q8 GGUF evaluation under llama.cpp, while the published 9B result was measured with an NVFP4 checkpoint under vLLM. Both use the GSM8K test split, but this table is a reported-score comparison rather than a perfectly controlled parameter-scaling experiment. The non-4B reference scores are reproduced from the 9B release card.
2.2 4B vs. 9B
Both students use the same DeepSeek-V4-Pro teacher pipeline and approximately 250,000-sample mathematics and STEM mixture. The 4B release prioritizes local efficiency; the 9B release retains more capacity for difficult multi-step STEM reasoning.
Metric
DeepSeek-V4-Pro-Qwen3.5-4B
DeepSeek-V4-Pro-Qwen3.5-9B
GSM8K
91.77%
94.50%
MMLU-Pro Math
80.40%
92.40%
MMLU-Pro Physics
74.20%
89.40%
MMLU-Pro Chemistry
74.80%
89.80%
MMLU-Pro Average
76.47%
90.53%
Interpretation: the 4B student remains strong on GSM8K, while Math is its strongest subject in the selected MMLU-Pro STEM sample. The 9B student retains higher scores across all three MMLU-Pro subjects. GSM8K and the reported 9B results use different inference builds, so this table should be read as a release overview rather than a controlled scaling experiment.
2.3 MMLU-Pro: Math, Physics & Chemistry
Each model in the comparison below was evaluated on 500 questions from each of three MMLU-Pro subsets—Math, Physics, and Chemistry—for 1,500 questions per model. The table follows the same column design and subject order as the 9B release card. The newly added Claude Mythos-distilled 27B result reaches 86.20% overall and appears directly before Claude Mythos-distilled 9B.
Model
Evaluation Build
Math
Physics
Chemistry
Average
DeepSeek-V4-Pro-Qwen3.5-4B
MTP-Q8 GGUF
80.40%
74.20%
74.80%
76.47%
DeepSeek-V4-Pro-Qwen3.5-9B
MTP-Q8_0 GGUF
92.40%
89.40%
89.80%
90.53%
Qwen3.5-9B
MTP-Q8_0 GGUF
90.60%
89.00%
89.20%
89.60%
Claude Mythos-distilled 27B
MTP-Q8 GGUF
88.40%
87.60%
82.60%
86.20%
Claude Mythos-distilled 9B
MTP-Q8_0 GGUF
86.00%
78.60%
78.60%
81.07%
Comparison note: all rows use 500 questions per subject, but the reported deployment configurations differ. The 4B run uses MTP-Q8 GGUF at 64K context, the 9B comparison rows use MTP-Q8_0 GGUF at 8K context, and Claude Mythos-distilled 27B uses MTP-Q8 GGUF at 32K context under llama.cpp.
⚙️ 3. Deployment Profile
🚀 Built for Practical Local Reasoning
The training adapter was merged with unsloth/Qwen3.5-4B in BF16. The reported deployment checkpoint is an MTP-enabled Q8 GGUF build using two draft tokens under llama.cpp speculative decoding.
The 4B scale is intended for users who value lower memory use and local inference accessibility. Exact speed and memory consumption depend on quantization, hardware, context length, batch size, and runtime configuration; no cross-hardware speed claim is made in this release.
🎯 4. Recommended Uses
Grade-school and general mathematical problem solving
Lightweight physics, chemistry, and broader STEM question answering
Structured reasoning and analytical instruction following
Local experiments with reasoning distillation and compact models
MTP-enabled llama.cpp deployment research
⚠️ 5. Limitations
This is an experimental 4B community model and remains subject to hallucinations, arithmetic mistakes, reasoning failures, and unstable behavior on difficult or underspecified tasks.
The reported MMLU-Pro result covers fixed 500-question samples from Math, Physics, and Chemistry rather than the complete subsets; broader categories remain unevaluated.
The GSM8K comparison combines reported results from different inference builds and backends; it is useful as a release overview, not as a perfectly controlled scaling study.
The model received no coding-specific SFT data, and coding or tool-use generalization has not yet been established for this 4B checkpoint.
A smaller parameter budget creates a visible gap versus the 9B release on the available MMLU-Pro STEM subsets.
Users should independently verify high-stakes mathematical, scientific, medical, legal, or factual outputs.
Acknowledgements: Kyle Hessling for hardware collaboration, training support, and evaluation assistance; DeepSeek for the teacher model and reasoning data source; Qwen for the base model family; Unsloth for efficient fine-tuning and conversion tooling; and the open-source inference community behind llama.cpp.
Citation:
@misc{jackrong_deepseek_v4_pro_qwen35_4b,
title = {DeepSeek-V4-Pro-Qwen3.5-4B},
author = {Jackrong},
year = {2026},
publisher = {Hugging Face}
}
