Views
No views yet
qwen3_5_moe, 35B MoE with a Qwen3-VL vision tower). Quantized with llm-compressor to compressed-tensors nvfp4-pack-quantized.--quantization flag (auto-detected).re:.*visual.*), the MoE routers (mlp.gate, mlp.shared_expert_gate), and lm_head. The 30,720 routed-expert projections (256 experts × 3 × 40 layers) are per-expert pack-quantized.1# recipe.yaml
2QuantizationModifier:
3 targets: [Linear]
4 ignore: [lm_head, 're:.*visual.*', 're:.*mlp.gate$', 're:.*mlp.shared_expert_gate$']
5 scheme: NVFP4| Benchmark | no-think | think |
|---|---|---|
| HumanEval+ (N=163) | 87.1% | 93.9% |
| MBPP+ (N=160) | 78.1% | 80.6% |
Reasoning-model eval tip: Ornith reasons at length. For one-shot code benchmarks (a) give it room (max_tokens ≥ 6500), and (b) extract the answer from after</think>— a naive code extractor that scans the whole message will grab draft code from inside the reasoning block and badly under-score the model.
| Config | single-stream | aggregate @ C=8 | aggregate (peak) |
|---|---|---|---|
| TP=2 | 114 tok/s | 466 tok/s | ~986 tok/s (saturates @ C=32) |
| TP=4 | 166 tok/s | 699 tok/s | ~2280 tok/s (still scaling @ C=64) |
--enforce-eager costs ~5× single-stream; the numbers above are with CUDA graphs on.)1vllm serve sakamakismile/Ornith-1.0-35B-NVFP4 \
2 --tensor-parallel-size 2 \
3 --max-model-len 8192 \
4 --disable-custom-all-reduce \
5 --trust-remote-code
6# env: NCCL_P2P_DISABLE=1 (no-NVLink hosts only)chat_template_kwargs: {"enable_thinking": true|false}.