1--attention-backend flashinfer # 2.7× vs triton at long context2--kv-cache-dtype fp8_e4m3 # 4-bit KV is 4× SLOWER3--speculative-algorithm EAGLE
4--speculative-num-steps 55--speculative-eagle-topk 16--speculative-num-draft-tokens 6# 5/1/6 — beat 2/1/3, 3/1/4, 4/1/5, 7/1/8, 9/1/10, 11/1/127--mamba-radix-cache-strategy extra_buffer_lazy
8--max-running-requests 19--max-mamba-cache-size 8# = requests × 8. See the coherence rule below.10--mamba-full-memory-ratio 0.9# do NOT raise. See below.11--page-size 3212--chunked-prefill-size 204813--mem-fraction-static 0.9714--reasoning-parser qwen3 --tool-call-parser qwen3_coder
🔴 Three traps that cost the most time
1. --max-mamba-cache-size must equal requests × 8
Hybrid GDN models reserve a recurrent-state pool. Each request needs 8 slots
(4 for the target with extra_buffer_lazy + 4 for the MTP draft).
Ask for more requests than your slots allow and SGLang silently clamps — the log
prints your requested value and then the clamped one, with no warning:
max_running_requests=4 ← what you asked for
max_running_requests=2 ← what you actually got
And concurrency is expensive. Measured, same battery, only this pair changing:
requests
slots
KV pool
short
63K
1
8
134,016
162.6
81.0
2
16
100,480
160.6
78.0
4
32
33,440
160.0
doesn't fit
Concurrency costs 75% of your context and returns nothing in single-stream speed.
2. --context-length does not reserve anything
It only declares. The number that matters is max_total_num_tokens, printed at startup.
--context-length 159776 ← what we declared
max_total_num_tokens 134016 ← what actually fits
Tell your client the second number. A client that believes the first will send a
request that fails at runtime instead of truncating cleanly.
3. --mamba-full-memory-ratio — the official value is wrong for long context
The SGLang playground raises it to 8.26 when MTP is on. Measured effect on our box:
ratio
KV pool
0.3
143,008
0.9 (default)
94,112
3.0
48,640
8.26 (official w/ MTP)
39,072
The official value cuts context by 58%. It is sized for many concurrent users, not
for a single long-context stream. Copying it without measuring halves your context.
Memory budget (measured at startup)
item
GB
note
Model weights (NVFP4)
18.59
MTP / speculative decoding
5.53
only 0.85 GB is weights; the rest is buffers
GDN state pool (8 slots)
1.94
~83 MB per slot
KV cache
3.88
free
0.68
34 KB per token of context. The MTP costs ~160K tokens of context — that is the
central trade of this model.
With MTP vs without (same battery, mem-fraction 0.90)
with MTP
without
context
62,880
272,800
short
165.0
74.2
63K
83.0
57.1
128K
doesn't fit
45.7
200K
doesn't fit
36.5
MTP roughly doubles speed and costs 4× context. Below ~130K it always wins.
⚠️ Without MTP you must drop to --mem-fraction-static 0.90. At 0.97 the pool grows so
large that no workspace is left and the first request dies with
Triton Error [CUDA]: out of memory — the server boots and answers /v1/models
perfectly, then breaks the moment it generates.
❌ Measured and rejected — do not retry
technique
why
--fp4-gemm-backend flashinfer_cudnn
−20% short, −30% at 63K. auto→cutlass is already best on SM120
--fp4-gemm-backend flashinfer_cutedsl
not supported on SM120 (it's SM100)
--attention-backend triton
+11% at 1K, −63% at 50K
--speculative-attention-mode decode
no gain, nominally worse at long context
--enable-fused-qk-norm-rope
zero gain
--speculative-adaptive
−21% at 63K, doubled variance
--speculative-algorithm NGRAM
3 attempts, 3 OOM — allocates outside the static pool
HiCache (--enable-hierarchical-cache)
crashes: 'HybridLinearKVPool' object has no attribute 'layer_num'
--enable-int8-mamba-checkpoint
−20% KV pool, no gain
--mamba-track-interval 64
nothing measurable
--tokenizer-worker-num 4
~890 MB VRAM per worker → OOM
--enable-mixed-chunk
silently ignored with spec decoding (server_args shows False)
--max-mamba-cache-size 24 / 32
24 → context drops to 12,288 · 32 → server hangs with no error
Deeper chains 7/1/8 … 11/1/12
accept len rises 3.39→3.73 while throughput falls 25%
KV 4-bit / TurboQuant / PolarQuant
4× slower; TurboQuant needs plain GQA, not hybrid
DSpark · DFlash · Weaver
all lost to the native MTP head
Quantizing the MTP head
SGLang forces quant_config=None for modelopt NVFP4; mtp.fc is a plain nn.Linear
🥇 The two biggest wins were not SGLang flags
Prompt field order — 12.9× less TTFT
The prefix cache matches on exact prefix. Anything volatile at the top of your system
prompt (session_id, turn counter, clock, current file, token count) changes every call
and invalidates everything after it.
layout (~50K stable context)
TTFT
volatile at the top
5.54 s
volatile at the end
0.43 s (steady state)
Same information, same tokens, same answer. Only the order changes. Costs nothing.
A debug flag in the gateway — +107%
LiteLLM running with --detailed_debug serializes and logs the whole payload per request.
With a 63K-token prompt that is ~250 KB formatted in Python before the model is called.
at 63K context
with
without
effective tok/s
39.5
81.8
streaming TTFT
2.87 s
1.00 s
Its own litellm_overhead_latency_metric reported 0.6 ms while the real cost was
2.1 s — the metric only wraps the LLM call. The signal is in
litellm_request_queue_time_seconds.
📏 Measurement protocol — skip this and you will measure noise
Five methodology errors produced five confident, wrong conclusions in one day.
Measure in your real regime. Free-form prose gives accept len 1.25; tool calls give
3.52. Same config. Every conclusion from the prose number was wrong.
Discard the first run after a restart. Measured bias: 12.7%. The same build read
79.2 cold and 89.3 in steady state.
n ≥ 3, report mean ± stdev. Within-run σ is ~3%; restart-to-restart is ~10%.
A difference is real only above ~9% (2.8σ). Anything below is the same number twice.
Client-side TTFT, never the server's input throughput. That metric divides by the
interval since the previous prefill log — in multi-turn chat it includes the last turn's
entire decode. Reading 96 tokens / 4.5 s as prefill duration produced a whole false theory.
A benchmark returning zero must fail loudly. One run printed a full table of 0.0 tok/s
with plausible TTFT — the prompt had overflowed the pool and the script counted zero chunks
without checking HTTP status.
Every new flag: drop --mem-fraction-static to 0.93 first. Three OOMs came from adding
a feature at 0.97. But then compare against a baseline at the same fraction — 0.93 costs
−30% at 63K on its own.
Files
file
what
scripts/00-preflight.sh
GPU / driver / docker / disk checks
scripts/01-download-model.sh
pulls the NVFP4 checkpoint
scripts/02-quantize-lmhead.py
lm_head BF16 → NVFP4 via modelopt
scripts/03-serve.sh
tuned launcher
scripts/bench/agentic_bench.py
throughput by workload type (prose / code / tool call / thinking)
scripts/bench/correctness.py
quality gate by objective correctness, not string diff
scripts/bench/multiturn_bench.py
long-context multi-turn
AGENTS.md
step-by-step for an AI agent to reproduce this
Open problem
Switching between two ~70K contexts costs 12.3 s of TTFT, and no memory configuration
changes it. Five configs from 112K to 143K of KV pool: 12.31–12.49 s, every time.
The log shows #cached-token: 0 with 43% of the pool free — the radix tree loses a
prefix it saw one turn earlier while having room to spare. Four hypotheses tested and
rejected (KV eviction, GDN state replay, general space, state-slot count).
Likely a limitation of SGLang's hybrid cache with alternating contexts, reinforced by
HiCache failing in the same HybridLinearKVPool. Unresolved — reported here rather
than explained away.
