DeepSeek-V4-Flash-0731-A100
Converted DeepSeek-V4-Flash-0731 weights for deployment on NVIDIA A100 / A800 (SM80) GPUs.
For deployment configuration, installation steps, launch scripts and benchmarks, see:
GitHub: yobo2u/DeepSeek-V4-Flash-0731-A100
SM90+ users (H100 / H800 / H20) should use the official upstream weights directly — this repository is not needed.
About This Conversion
The SM80 architecture used by A100/A800 does not support the native FP8 execution path that the original model targets on newer GPUs. The weights in this repository were converted offline so they can run on SM80 GPUs together with a dedicated SGLang 0.5.16 monkeypatch:
- Non-expert weights: converted to BF16
- MoE expert weights: kept in MXFP4
- Shards: 48 safetensors files
- Total size: approximately 173 GB (decimal)
- Context length: up to 1,048,576 tokens
- Recommended inference framework: SGLang 0.5.16
- Recommended hardware: 8× A100 80GB or 8× A800 80GB, TP=8
- Speculative decoding: DSpark supported
This repository provides converted model weights only. The full runtime environment, pinned versions, monkeypatch source, launch arguments and benchmark scripts are maintained in the GitHub configuration repository.
Deployment
Do not reuse launch commands written for stock vLLM or newer-architecture GPUs. SM80 deployment requires a pinned SGLang version plus the A100 monkeypatch:
Download
1hf download yobo2u/DeepSeek-V4-Flash-0731-A100 \
2 --local-dir /path/to/models/DeepSeek-V4-Flash-0731-A100
For a model of this size, use low download concurrency and resumable transfers, then verify that all 48 shards referenced by model.safetensors.index.json are present.
The same weights are also published on ModelScope:
yobo2u/DeepSeek-V4-Flash-0731-A100Verified Configuration
| Item | Value |
|---|
| GPU | 8× NVIDIA A800-SXM4-80GB (SM80) |
| Tensor parallel | 8 |
| SGLang | 0.5.16 (commit fdebc938) |
| Dtype | BF16 + MXFP4 experts |
| Serving port | 8082 |
| Context length | 1,048,576 |
--mem-fraction-static | 0.85 |
--chunked-prefill-size | 32,768 |
--max-running-requests | 16 |
| Speculative decoding | DSpark |
Measured Performance
Measured on the verified configuration above (config C), covering 1K / 4K / 16K / 64K / 128K context × concurrency 1 / 8 / 16 × 2 full repeats = 30 groups, 900 requests, 0 errors and 0 timeouts:
| Metric | Value |
|---|
| Single-stream decode (concurrency 1, 1000/TPOT) | ~217 tok/s |
| Single-stream aggregate throughput (concurrency 1) | ~205 tok/s (166–222) |
| Aggregate throughput (concurrency 16) | ~1,232 tok/s (peak 1,334) |
| DSpark accept rate | ~60% (0.40–0.86, increases with context) |
| DSpark accept length | ~4.01 (3.01–5.30) |
| TTFT (concurrency 1) | ~296 ms |
| Peak VRAM | ~51.9 GB/GPU |
Accept rate and accept length rise substantially with context length: about 0.40 / 3.0 at 1K, and about 0.82 / 5.1 at 128K.
The table reports means across all 30 groups; per-group values and raw data are in the JSON files under the GitHub benchmarks directory.
Tuning Findings
A three-way factor decomposition over mem-fraction-static and chunked-prefill-size (30 groups / 900 requests each):
| Config | mem-fraction | chunked-prefill | Mean throughput | Peak VRAM |
|---|
| A | 0.85 | 16,384 | 746.2 tok/s | 48.58 GiB |
| B | 0.90 | 32,768 | 759.8 tok/s | 53.80 GiB |
| C (recommended) | 0.85 | 32,768 | 766.4 tok/s | 51.92 GiB |
- Keep
mem-fraction-static at 0.85. Raising it to 0.90 squeezes the KV cache headroom, dropping the 128K accept rate from 0.80 to 0.69 and costing roughly 11% throughput at long context.
- Raising
chunked-prefill-size from 16,384 to 32,768 helps at short-to-medium context and higher concurrency, and is not the cause of the 128K regression.
- Full factor-decomposition report (HTML and PDF) is available in the GitHub benchmarks directory.
Performance varies with input length, concurrency, driver, CUDA version, interconnect topology and the specific A100/A800 SKU. The GitHub repository is the source of truth for methodology and results.
Limitations and Notes
- These are community weights converted for SM80 deployment — not an official DeepSeek model release.
- Running them depends on a dedicated monkeypatch. Downloading the weights alone does not make them loadable by Transformers, vLLM, or unmodified SGLang.
--quantization fp8 is part of how the current SGLang monkeypatch path is wired up; it does not mean every weight is stored in native FP8.- Follow the version matrix and launch arguments in the GitHub documentation to avoid SGLang API and kernel incompatibilities.
- For model capabilities, recommended sampling parameters, chat template format and usage policy, refer to the original model repository.
Credits
License
Released under the MIT License, following the original repository. Users must also comply with the upstream model repository's license terms and applicable laws and regulations.
