Asymmetric AWQ 4-bit (W4A16, group size 128) quantization of deepreinforce-ai/Ornith-1.0-35B (qwen3_5_moe — fine-grained MoE, hybrid linear/full attention, omni-multimodal).

Produced in-house with AutoRound using the RTN (round-to-nearest, iters=0) algorithm in asymmetric mode:

auto-round-mllm \
  --model deepreinforce-ai/Ornith-1.0-35B \
  --scheme W4A16 --algorithm rtn --asym --group_size 128 \
  --format auto_awq

Why RTN, and why this repo exists

This checkpoint is the exact artifact deployed in production on 8×V100-SXM2 (sm_70) served by 1Cat-vLLM via its awq_sm70_moe (turbomind) MoE path — the one 4-bit MoE format with real Volta kernels.

In an A/B test, RTN beat the pile-10k-calibrated AutoRound-AWQ variant: the calibrated/tuned build measurably degraded reasoning and code quality, so the simpler RTN quantization was chosen for serving. A separate repo, rapatel0/Ornith-1.0-35B-AWQ-asym, holds a different (non-production) AWQ build; this repo is the checkpoint actually in production.

Serving notes (1Cat-vLLM on V100)

--quantization awq (auto-routes to the awq_sm70_moe turbomind path on sm_70)
--dtype float16, --tensor-parallel-size 1 (scale by TP1 replica fan-out, not batch — fine-grained MoE batches poorly), --max-num-seqs 1
fp16 KV cache, --max-model-len 131072
chat_template.jinja is included in this repo (the base model's template).
~91 tok/s single-slot decode.

Includes quantization_config.json (sym: false, quant_method: awq, provider: auto-round, group_size: 128).

rapatel0/Ornith-1.0-35B-RTN-AWQ-asym

Ornith-1.0-35B — RTN asym-AWQ (W4A16, g128) for V100 / sm_70

Why RTN, and why this repo exists

Serving notes (1Cat-vLLM on V100)

rapatel0/
Ornith-1.0-35B-RTN-AWQ-asym