DeepSeek-R1-FlagOS-Cambricon-BF16 provides an all-in-one deployment solution, enabling execution of DeepSeek-R1 on Cambricon DSAs. As the first-generation release for the Cambricon series, this package delivers three key features:
Open-source inference execution code, preconfigured with all necessary software and hardware settings.
Pre-built Docker image for rapid deployment on Cambricon.
High-Precision BF16 Checkpoints:
BF16 checkpoints dequantized from the official DeepSeek-R1 FP8 model to ensure enhanced inference accuracy and performance.
Consistency Validation:
Evaluation tests verifying consistency of results between NVIDIA H100 and Cambricon-MLU590.
Technical Summary
Serving Engine
We use FlagScale as the serving engine to improve the portability of distributed inference.
FlagScale is an end-to-end framework for large models across multiple chips, maximizing computational resource efficiency while ensuring model effectiveness. It ensures both ease of use and high performance for users when deploying models across different chip architectures:
One-Click Service Deployment: FlagScale provides a unified and simple command execution mechanism, allowing users to fast deploy services seamlessly across various hardware platforms using the same command. This significantly reduces the entry barrier and enhances user experience.
Automated Deployment Optimization: FlagScale automatically optimizes distributed parallel strategies based on the computational capabilities of different AI chips, ensuring optimal resource allocation and efficient utilization, thereby improving overall deployment performance.
Automatic Operator Library Switching: Leveraging FlagScale's unified Runner mechanism and deep integration with FlagGems, users can seamlessly switch to the FlagGems operator library for inference by simply adding environment variables in the configuration file.
Triton Support
We validate the execution of DeepSeed-R1 model with a Triton-based operator library as a PyTorch alternative.
We use a variety of Triton-implemented operation kernels—approximately 70%—to run the DeepSeek-R1 model. These kernels come from two main sources:
Most Triton kernels are provided by FlagGems (https://github.com/FlagOpen/FlagGems). You can enable FlagGems kernels by setting the environment variable USE_FLAGGEMS. For more details, please refer to the "How to Run Locally" section.
Also included are Triton kernels from vLLM, including fused MoE.
BF16 Dequantization
We provide dequantized model weights in bfloat16 to run DeepSeek-R1 on Cambricon DSAs, along with adapted configuration files and tokenizer.
Bundle Download
Requested by Cambricon, the file of docker image and model files should be applied by email.
Usage
Cambricon
Basic Image
basic software environment that supports model running
ecosystem@cambricon.com Contact by email，please indicate the unit/contact person/contact information/equipment source/specific requirements
Model
model weight and configuration files
ecosystem@cambricon.com Contact by email，please indicate the unit/contact person/contact information/equipment source/specific requirements
Evaluation Results
Benchmark Result
Metrics
DeepSeek-R1-H100-CUDA
DeepSeek-R1-FlagOS-Cambricon-BF16
GSM8K (EM)
95.75
95.15
MMLU (Acc.)
85.34
85.61
CEVAL
89.00
89.38
AIME 2025(Pass@1)
76.67
73.33
GPQA-Diamond (Pass@1)
70.20
73.23
MATH-500 (Pass@1)
93.20
92.60
How to Run Locally
📌 Getting Started
Environment Setup
bash
1# install FlagScale2git clone https://github.com/FlagOpen/FlagScale.git
3cd FlagScale
4pip install.56# download image and ckpt7flagscale pull --image <IMAGE> --ckpt deepseek-ai/DeepSeek-R1 --ckpt-path /nfs/DeepSeek-R1
89# Note: For security reasons, this image does not have passwordless configuration. In multi-machine scenarios, you need to configure passwordless access for the image yourself.1011# build and enter the container12docker run -e --net=host --pid=host --ipc=host -v /tmp/.X11-unix:/tmp/.X11-unix --privileged -it -v /nfs:/nfs -v /opt/data/:/opt/data/ -v /usr/bin/cnmon:/usr/bin/cnmon --name flagrelease_cambricon <IMAGE> /bin/bash
Download and install FlagGems
bash
1git clone https://github.com/FlagOpen/FlagGems.git
2cd FlagGems
3git checkout deepseek_release_cambricon
4# no additional dependencies since they are already handled in the Docker environment5pip install ./ --no-deps
6cd../
Download FlagScale and unpatch the vendor's code to build vllm
bash
1git clone https://github.com/FlagOpen/FlagScale.git
2cd FlagScale
3git checkout ae85925798358d95050773dfa66680efdb0c2b28
4# please set the name and email in git config in advance, for example: git config --global user.name "your_name"; git config --global user.email "your_email"5python tools/patch/unpatch.py --device-type cambricon_MLU --commit-id 57637057 --dir build
6cd build/cambricon_MLU/FlagScale/vllm
7pip install -e . -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
8cd vllm_mlu
9pip install -e . -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
10cd../../
Serve
bash
1# config the deepseek_r1 yaml2build/cambricon_MLU/FlagScale/
3├── env.sh
4├── examples/
5│ └── deepseek_r1/
6│ └── conf/
7│ └── config_deepseek_r1.yaml # set hostfile, env.sh absolute path and ssh_port(optional), if it is passwordless access between containers, the docker field needs to be removed8│ └── serve/
9│ └── deepseek_r1.yaml # set model parameters and server port10# install flagscale11pip install.12# serve13flagscale serve deepseek_r1
Contributing
We warmly welcome global developers to join us:
Submit Issues to report problems
Create Pull Requests to contribute code
Improve technical documentation
Expand hardware adaptation support
📞 Contact Us
Scan the QR code below to add our WeChat group
send "FlagRelease"
