Escha Runtime — qwen3dense

Compatible models

This runtime targets the qwen3_5 dense architecture. Its wheel also happens to register the eschamoe mixture-of-experts method, so a qwen3_5_moe model will load too — but the tuning, the defaults in sglang/serve.sh and the documentation here are all written for the dense architecture. For a mixture-of-experts model use escha-runtime-qwen3moe, whose defaults are measured on it. A model of a genuinely different architecture will not load — use the matching escha-runtime-<arch> repo.

Quickstart

pip install "torch==2.9.*" is a hard pin, not a suggestion. A bare torch>=2.9 resolves to a newer minor and import escha then fails with undefined symbol: _ZN3c10... — the compiled extension is ABI-linked to libtorch, and that ABI is not stable across PyTorch minors.

Thinking, and why you probably want a budget

Requirements

• NVIDIA GPU, compute capability 8.0–12.0 (Ampere → Blackwell), Linux x86-64 with glibc ≥ 2.28. The kernel launch route auto-selects per GPU; you never set it. Per-architecture and per-VRAM launch recipes: sglang/INSTALL.md → Running on your GPU.
• Python 3.12 (the wheel is cp312-only) + CUDA-12 PyTorch 2.9.x. The wheel handles every other dependency.
• A working host C compiler and Python dev headers. Triton JIT-compiles a small shim at CUDA-graph capture time — this is separate from ptxas and from a CUDA toolkit, so "driver only" does not cover it. On slim container images a stripped libisl breaks cc1 while gcc --version still succeeds, and the failure surfaces ~40 s in as a gcc CalledProcessError inside cuda_graph_runner.py — which reads like a runtime bug and is not. Preflight in sglang/INSTALL.md.
• 24 GB VRAM for the shipped defaults (65,536-token context, ~8–9 concurrent streams at short prompts) with a ~10.15 GB model. Those two figures are not simultaneous: the default pool is 68,686 tokens, which is one full-length 64k request or ~8 requests of ~8k. Note also the stream ceiling: MAMBA_RATIO=0.3 sizes the recurrent-state pool, which clamps max_running_requests to 8–9 on a 24 GB card, so the 12/16 entries in the default CUDA_GRAPH_BS are dropped and never captured. To serve more streams raise MAXREQ/MAXMAMBA with MEM — the throughput recipe is in the model card; for long context (128k measured on a 24 GB card) see By VRAM. 16 GB is now measured too, on an RTX 5060 Ti — ~30 tok/s single-stream, and far more context than we had guessed: 110,592 tokens, with an fp8 KV cache doing most of that work. See The 16 GB tier.

Changelog

Single-GPU users are unaffected. Every new code path is gated on world_size > 1; at --tp-size 1 the loader is byte-for-byte what 1.1.1 did. Verified as an identical shard layout and byte-identical greedy output.

TP > 1 is new and lightly tested — treat it as experimental. It was contributed and validated by @ginerJuanUdesa on 2× RTX 3090 (symmetric 6.02 GB/rank, coherent greedy output). We have one GPU and could not reproduce it, and no numerical equivalence check against --tp-size 1 has been run yet. If you use it for evaluation, sanity-check a benchmark against the single-GPU numbers first. Note that a multi-rank all-reduce reorders float accumulation, so TP > 1 output is not expected to match TP = 1 bit-for-bit even when correct.

On Ampere/Ada/Hopper you can add DETERMINISTIC=1 to remove that reduction-order variance if you want a stricter comparison.

Known limitations

• Serving throughput has been measured on four cards — RTX 5090 (32 GB, sm_120), RTX 4090 (24 GB, sm_89), RTX 3090 (24 GB, sm_86) and RTX 5060 Ti (16 GB, sm_120); all but the 4090 by independent evaluators working only from these docs. The 4090 is also where the model's thinking-on benchmarks (GPQA-Diamond, LiveCodeBench) were produced; its thinking-off commonsense suite ran on an L40. The 40 GB+ tier in the cookbook is still configuration guidance derived from the model size and the wheel's architecture coverage, not a measurement. Per-GPU numbers live on the model card.
• On Ampere the auto-selected kernel route is the slower one at batch 1. ESCHA_ROUTE resolves to lovelace on sm_80/sm_86, but forcing ESCHA_ROUTE=blackwell measured 1.72× faster single-stream on an RTX 3090 (23.6 → 40.7 tok/s, TPOT 42.4 → 24.6 ms) with identical output. The two routes are bit-identical launch geometries, so this is safe to set; the gain is batch-1-only (parity at 2–16). Serving one user on Ampere? Set it.
• DETERMINISTIC=1 fails on consumer Blackwell (sm_120). The deterministic attention kernel requests 104 KB of shared memory per block, above the sm_120 limit, and the server exits during startup. It works on Ampere, Ada and Hopper.
• Greedy output is not bit-reproducible across requests. Batch composition changes fp16 accumulation order, so a near-tie can flip and a long reasoning chain diverges from there. Two identical greedy requests may return different text. Use DETERMINISTIC=1 when you need reproducibility, and never A/B two configurations by diffing one generation.
• CUDA-graph batch sizes are capped at 32, the decode kernel's maximum M (torch.ops.escha.escham_decode_gemv_max_m()). The shipped default list stops at 16 because that is where aggregate throughput peaks on a 4090; capture at 24/32 works and is worth it if you serve that many streams. Past 32 a batch falls through to a large-M path meant for prefill, so the runtime refuses to capture it rather than bake in the wrong kernel.
• ATTN_BACKEND=triton is required on consumer Blackwell (RTX 50-series). The default flashinfer backend asserts on this hybrid architecture at sm_120. The assertion names three acceptable backends — triton, trtllm_mha, fa4 — of which only triton has been run on this model. Note that sm_120 shows steeper long-prompt decode decay than sm_89 (88.5% vs 96.3% of short-prompt rate at a 5,000-token prompt); the attention path is the obvious suspect and nobody has run the A/B that would confirm it.
• The model's stock chat template rejects the developer role. It accepts system, user, assistant and tool; anything else raises Unexpected message role. and surfaces as an HTTP 400 from the template, not as a server fault. Several OpenAI-compatible agent harnesses send role: "developer" for their system prompt and so fail every request — map it to system client-side, or patch the template. The same template also raises on genuinely malformed conversations (a missing user query, a system message in the wrong position) and on a reasoning_effort outside xhigh/medium/low. All of this is the base model's own template, carried over unmodified.
• One report of a GPU falling off the bus under sustained load, on a passed-through card. @JacksonAtkinsX hit NVRM: Xid ... 79, GPU has fallen off the bus four times out of four on an RTX 5060 Ti 16 GB, 12–18 minutes into a sustained single-stream agent workload, and each time the card needed a full host power cycle — FLR, secondary-bus reset and remove/rescan all reported success while the device stayed in D3cold. It reproduced across two driver branches, two runtime versions, both KV dtypes and three context lengths, in prefill and in decode alike. The card is passed through to a VM with vfio.
  We have not reproduced anything like it on the 3090, 4090, 5090 or L40 across the campaigns behind these docs, none of them on a passed-through GPU. Xid 79 is a bus-level fault: a misbehaving CUDA kernel produces an Xid 13/31/43 or an MMU fault and leaves the device resettable, whereas here no reset short of a power cycle worked, which is a platform signature rather than a userspace one. Two hypotheses that report could not close, and what we would check:
  • Run any other sustained 100% load for 30 minutes on the same box — gpu-burn, or a different model on a stock runtime. This is the one test that settles it: if that also falls off the bus, the runtime is not involved.
  • Power transients. nvidia-smi was sampled at 5 s, which cannot see the sub-millisecond spikes that trip a PSU's over-current protection, so "166 W under a 180 W cap with no throttle flags" does not rule power out. Log power.draw.instant and temperature.memory at 100 ms, and check PSU headroom on the rail feeding the card.
  • PCIe power management under vfio, which the D3cold-and-never-returns behaviour fits: vfio-pci disable_idle_d3=1, and pcie_aspm=off pcie_port_pm=off on the host command line.
  One thing we will say on our own side: this runtime's 2-bit decode path is arithmetic-heavy and holds a high, flat ALU duty cycle rather than the burstier profile of an fp16 matmul, so it is a harsher electrical stressor than most inference workloads. That is a reason to want power headroom before a long unattended run, not evidence of a defect.

License