Innovation and Open source are the best tributes and reproductions of DeepSeek.
The open-source ethos, rooted in technological equity, upholds two key principles: free access for all developers and the opportunity for technical contributions. While DeepSeek exemplifies the first principle, the second principle is hindered by restrictive training strategies, unclear data sources, and the high costs of model training. These constraints limit the open-source community's capacity to contribute and impede technological progress.
To overcome these challenges, we launched a comprehensive reproduction project of DeepSeek. This initiative involves training models from scratch, replicating DeepSeek's architecture and algorithms. We will fully open-source the training code, datasets, and models, offering code framework, reference solution, and base models for low-cost continual exploration.
🌈 Update
[2025.03.11] TinyDeepSeek repo is published！🎉
Reproduction of DeepSeek-R1
Architecture
Click to expand
arch
As shown in above Figure, the DeepSeek technical report introduces three architectural designs:
A. Multi-Head Latent Attention (MLA)
B. Load Balancing Strategy without Auxiliary Loss: Please refer code for implementation.
C. Multi-Token Prediction: Please refer code for implementation.
Our Detailed architectural parameter are shown in Table below.
arch
Data Construction
Click to expand
arch
Pretrain:
Meta & Labeled Data: TBD
Process Code: Please refer to Link for implementation.
