Ornith-1.5-35B-A3B with DFlash in SGLang
This repository contains no model weights. It documents a recipe for serving
Ornith-1.5-35B-A3B with the
Qwen3.5-35B-A3B-DFlash draft model
in
SGLang.
Results
Tested on two A100 80GB with single-stream greedy decoding and 640-token
generations. GSM8K answers were checked numerically. HumanEval solutions were
run against their unit tests.
| configuration | decode tok/s | speedup | accepted tokens (GSM8K / HumanEval) | GSM8K | HumanEval |
|---|
| Base model | 202.5 | 1.00x | — | 93.3% | 87.5% |
| DFlash, block size 16 | 470.2 | 2.32x | 6.1 / 7.7 | 90.0% | 90.0% |
The speedup is smaller than the dense 9B model sees (about 4.6x) because this
sparse MoE only activates about 3B parameters per token, so its baseline is
already fast (202 vs 45 tok/s). The draft still gets 6 to 8 tokens accepted per
step, which is where the 2.3x comes from, and code speeds up more than math
(about 2.7x versus 2.1x). The task pass rates stayed close to the baseline in
this sample. The benchmark does not claim identical text across separate server
runs.
Use lmsysorg/sglang:v0.5.17-cu129. Keep transformers at 5.12.1 with SGLang
0.5.17. The bf16 weights are about 73 GB, so how many GPUs you need depends on
how much VRAM you have.
On two 80 GB GPUs
This is the tested setup. The weights plus the DFlash recurrent-state cache do
not fit on a single 80 GB card, so split across two:
1python -m sglang.launch_server \
2 --model-path ornith-ai/Ornith-1.5-35B-A3B \
3 --speculative-algorithm DFLASH \
4 --speculative-draft-model-path z-lab/Qwen3.5-35B-A3B-DFlash \
5 --speculative-dflash-block-size 16 \
6 --reasoning-parser qwen3 \
7 --tp 2 \
8 --mem-fraction-static 0.80 \
9 --max-total-tokens 40960 \
10 --max-running-requests 4 \
11 --mm-feature-transport cpu \
12 --trust-remote-code
On one GPU with 96 GB or more
If you have a single card with at least about 90 GB (a 96 GB RTX 6000 Pro, an
H200, and so on), drop the tensor parallelism and run it on one GPU. The 73 GB of
weights leave enough room for the draft, the state cache, and the CUDA graphs:
1python -m sglang.launch_server \
2 --model-path ornith-ai/Ornith-1.5-35B-A3B \
3 --speculative-algorithm DFLASH \
4 --speculative-draft-model-path z-lab/Qwen3.5-35B-A3B-DFlash \
5 --speculative-dflash-block-size 16 \
6 --reasoning-parser qwen3 \
7 --tp 1 \
8 --mem-fraction-static 0.85 \
9 --max-total-tokens 40960 \
10 --max-running-requests 4 \
11 --mm-feature-transport cpu \
12 --trust-remote-code
A single 80 GB card is not enough for the bf16 weights: at ~73 GB they leave too
little for the state cache and graphs, which is where an early attempt failed.
For a single smaller card you would need an FP8 checkpoint on a Hopper GPU (the
FP8 MoE kernel needs sm90+, so it will not run on an A100), or just use the two
GPU setup above.
The last four flags in both commands matter on this model. --max-total-tokens
and --max-running-requests keep the KV pool and the mamba state cache small
enough that the CUDA graphs still fit; without them the draft graph capture runs
out of memory. --mm-feature-transport cpu avoids the pidfd_getfd syscall that
some hosts block for the vision feature transport (this run sends only text).
Full reproduction files are in the
GitHub repository.
Attribution
This recipe and benchmark were prepared by ultimatechris. The models keep
their own licenses.
