DeepSeek-V4-Flash-0731 DSpark Graph-8 on two GB10 nodes
A reproducible deployment and benchmark package for the official
deepseek-ai/DeepSeek-V4-Flash-0731
checkpoint on two NVIDIA GB10-class nodes using vLLM tensor parallelism, DSpark speculative decoding, NVFP4 DS-MLA KV cache, FlashInfer B12X MoE, RoCE, and ordinary CUDA Graph capture through batch shape 8.
Independent community recipe. This repository is not an official DeepSeek release and contains no model weights. Download the official checkpoint from DeepSeek-AI. DSpark accelerates target-verified generation; it does not change the target model's knowledge or semantics.
Headline result
Stabilized balanced profile
The current safer production profile runs
two concurrent sequences with
GPU_MEMORY_UTILIZATION=0.75, 4,096 batched tokens, Graph-8, and the UCX
registration-cache safeguards documented in
docs/STABILITY.md.
In a simultaneous two-request proof, both requests stayed scheduled
(
running=2,
waiting=0) and each generated 1,005 exact bounded tokens in
21.05 seconds:
| Concurrency | Per-stream decode | Aggregate decode | Post-baseline allocation errors |
|---|
| 2 | 47.74 tok/s | 95.48 tok/s | 0 on both nodes |
At the end of that proof the nodes retained approximately 20 GiB and 18 GiB
MemAvailable, memory PSI avg10 was 0.00, and the service had zero restarts.
This is the recommended default profile; it does not imply two simultaneous
full-1M-token requests because the KV pool is shared.
Initial warm single-request qualification
On a two-node production pair, three warm single-request repetitions per shape produced:
| Workload | Median decode | Median TTFT | DSpark acceptance |
|---|
| Count 1–300, exact bounded output | 91.89 tok/s | 0.186 s | 98.81% |
| Twenty Python dataclasses | 84.43 tok/s | 0.249 s | 98.66% |
| Low-predictability technical prose | 40.08 tok/s | 0.235 s | 32.10% |
These medians came from the initial Graph-8 qualification profile before the
headroom hardening above. The replaced GGUF/llama.cpp lane delivered about
9.3 tok/s on the same two-node class, so the measured uplift was approximately
9.9×, 9.1×, and 4.3× respectively. These are workload-specific measurements,
not a universal throughput floor, and should not be compared directly with the
two-stream aggregate measurement.
Graph-8 improved matched decode over Graph-4 by:
| Workload | Graph-4 | Graph-8 | Gain |
|---|
| Count 1–300 | 74.28 | 93.90 tok/s | +26.42% |
| Dataclasses | 75.22 | 86.33 tok/s | +14.78% |
| Technical prose | 32.12 | 40.87 tok/s | +27.24% |
The K5 proposer verifies six-token shapes. Capturing [1,2,4,8] rather than stopping at 4 covers that shape.
Qualification gates
- Official checkpoint revision:
9e165c30e2704aec5d9d593cce3eebd58bbef1cb
- 48/48 safetensor shards verified independently on both ranks
- 96/96 draft parameters loaded; no skipped tensors
- Configured context: 1,048,576 tokens
- Stabilized-profile KV capacity observed at startup: 1,372,579 tokens (1.31× the configured full-context ceiling, shared across active requests)
- Stabilized two-request proof: 47.74 tok/s per stream, 95.48 tok/s aggregate,
running=2, waiting=0
- Stabilized post-proof headroom: approximately 20 GiB / 18 GiB MemAvailable, memory PSI
avg10=0.00
- Production exact uncached retrieval: 159,979 prompt tokens, 97.50 s, cached tokens 0
- Equivalent pre-promotion qualification pair: 949,896 prompt tokens, exact key retrieval, 1,057.98 s TTFT, cached tokens 0
- Native OpenAI-format tool call: passed
- Tool-result continuation: passed
- Closed-world strict JSON schema: passed
- Production service restart and post-restart smoke: passed
One unconstrained prose parity sample differed by one output token between speculative and non-speculative runs (256 vs. 255 tokens). Bounded outputs, retrieval keys, tool names/arguments, and strict JSON remained exact. Do not interpret this package as a claim of byte-identical unconstrained prose.
Canonical runtime profile
1Model: DeepSeek-V4-Flash-0731
2Tensor parallel: 2 nodes
3DSpark draft tokens: 5 (K5)
4KV cache: nvfp4_ds_mla
5MoE backend: flashinfer_b12x
6VLLM_USE_BREAKABLE_CUDAGRAPH=0
7CUDA graph capture set observed: [1, 2, 4, 8]
8--max-cudagraph-capture-size 8
9--max-model-len 1048576
10--max-num-seqs 2
11--max-num-batched-tokens 4096
12--gpu-memory-utilization 0.75
13--block-size 256
14UCX_MEM_MMAP_HOOK_MODE=none
15UCX_RCACHE_MAX_UNRELEASED=1024
16RoCE v2 / NCCL IB
Pinned runtime image:
1ghcr.io/anemll/dspark-vllm-gx10:0.1.1
2registry digest: sha256:a83948492cf13df455170fb42885f5ef4db54fefe0feff0f841ecbff464ac9d8
3validated local image ID: sha256:3430d6614a8e2925f34d059af6caf05aff42387326db4d05639a60f10f2654d8
Quick start
1. Prepare both nodes
Each node needs:
- one GB10-class CUDA GPU;
- Docker with Compose and NVIDIA Container Toolkit;
- RoCE-capable connectivity visible under
/dev/infiniband;
- the complete official 0731 checkpoint at the same local path;
- passwordless SSH only if you add your own coordinator wrapper.
Verify all 48 model shards against a trusted manifest before launch.
2. Configure node-specific environments
Copy runtime/.env.example to .env.worker and .env.head. Set a different VLLM_HOST_IP and NCCL_IB_GID_INDEX on each node. MASTER_ADDR must be the head node's fabric address on both ranks.
Do not reuse one GID index across nodes without checking sysfs: RoCEv2 GID indices can differ and can move after reboot or link changes.
Do not restore the earlier
0.80 GPU-utilization / 8,192-batched-token defaults
without a fresh pressure-qualified experiment. Read
docs/STABILITY.md
before changing memory, sequence, graph, or watchdog settings.
3. Start worker, then head
Worker rank:
1COMPOSE_DISABLE_ENV_FILE=1 NODE_RANK=1 HEADLESS=1 \
2 docker compose --env-file runtime/.env.worker \
3 -f runtime/docker-compose.yml -p deepseek-v4-flash-0731 up -d
Head rank:
1COMPOSE_DISABLE_ENV_FILE=1 NODE_RANK=0 HEADLESS= \
2 docker compose --env-file runtime/.env.head \
3 -f runtime/docker-compose.yml -p deepseek-v4-flash-0731 up -d
Keep the API loopback-only unless you deliberately provide authentication and firewalling. If remote clients need access, put an authenticated reverse proxy in front of vLLM.
4. Verify
1BASE_URL=http://127.0.0.1:8000 MODEL=deepseek-v4-flash-0731 \
2 python3 scripts/production_proof.py --context-tokens 160000
3
4BASE_URL=http://127.0.0.1:8000 MODEL=deepseek-v4-flash-0731 \
5 python3 scripts/benchmark.py --repetitions 3 --output benchmark.json
Repository contents
runtime/docker-compose.yml — two-node vLLM/DSpark Graph-8 profileruntime/.env.example — node-specific configuration templatescripts/production_proof.py — model, tools, strict JSON, and long-context gatesscripts/benchmark.py — streaming TTFT/decode/acceptance harnessbenchmarks/production-summary.json — sanitized measured resultsdocs/METHODOLOGY.md — definitions, test conditions, caveats, and parity disclosuredocs/STABILITY.md — OOM/headroom incident, stabilized defaults, startup-baseline rule, and promotion gatesNOTICE — upstream projects and pinned revisions
Attribution
This package builds on the official DeepSeek model and community GB10 work from Anemll, MiaAI-Lab, Tony Deangelo, and Weschera. Exact repositories and commits are recorded in NOTICE.
License
Code and documentation in this package are MIT-licensed. The official model and weights are separately provided by DeepSeek-AI under MIT; obtain them from the upstream repository. Preserve all upstream notices when redistributing upstream code.
