Ornith-1.0-35B — W8A16 AutoRound (INT8 weight-only)
This is an unofficial W8A16 (8-bit weights, 16-bit activations) quantized version of
deepreinforce-ai/Ornith-1.0-35B, created with
AutoRound.
Ornith-1.0-35B is DeepReinforce AI's lightweight agentic-coding model.
Quantizing the routed experts and attention projections to INT8 shrinks the checkpoint from ~70 GB (BF16) to ~38 GB, so the model fits comfortably on 2×24 GB GPUs while keeping the
output distribution very close to the original.
In addition to the quantization, this checkpoint bundles a re-trained MTP draft module for speculative decoding. The MTP weights originate from
Qwen/Qwen3.5-35B-A3B and were fine-tuned against Ornith's outputs. See
MTP draft module for more.
What is quantized
INT8 (per-output-channel, symmetric) is applied to the routed-expert MLPs (gate_up_proj, down_proj) and the full-attention projections. The following are kept at BF16:
embed_tokens, lm_head, the MoE router (mlp.gate), the shared expert (shared_expert), the linear-attention / gated-delta mixers (linear_attn), and the entire vision tower (visual).
In total ~30,760 / 31,181 linear modules are quantized. the rest stay BF16.
Quantization details
| Field | Value |
|---|
| Base model | deepreinforce-ai/Ornith-1.0-35B |
| Method | AutoRound (intel/auto-round) |
| Scheme | W8A16 |
| Bits | 8 |
| Group size | -1 (per-output-channel) |
| Symmetric | yes |
| Format | auto_round (gptq-style packing) |
| Unquantized layers | embed_tokens, lm_head, mlp.gate, shared_expert, linear_attn, visual |
| Calibration data | 25 % NeelNanda/pile-10k + 75 % codeparrot/github-code-clean |
| Calibration samples | 1024 (256 pile + 768 github-code) |
| Iterations | 1000 |
| Batch size | 8 |
| Sequence length | 2048 |
| GPU used for quant | 2× RTX 3090 |
KLD results
Quality was verified by measuring the KL divergence of the next-token distribution against the original BF16 model, KL(P_bf16 ‖ Q_int8), over 131,072 tokens (128 passages × 1024 tokens from
NeelNanda/pile-10k, held out from calibration). Lower is better.
| Metric | Value |
|---|
| Mean KL | 0.00348 nats |
| Median KL | 0.00139 nats |
| 99th-percentile KL | 0.0321 nats |
Reverse KL KL(Q‖P) | 0.00354 nats |
| Top-1 agreement | 97.5 % |
MTP draft module
This checkpoint additionally ships a MTP draft module (model-mtp.safetensors) for speculative decoding.
This module is not part of the official Ornith release. It was rebuilt and re-trained as described below.
MTP details
-
The MTP module here was initialized from the mtp.* tensors of the original Qwen/Qwen3.5-35B-A3B checkpoint and then fine-tuned to match Ornith-1.0-35B's output distribution (self-distillation, see below).
-
This is an unofficial community artifact. It is not affiliated with or endorsed by DeepReinforce AI or the Qwen team.
Training recipe (self-distillation)
- Data: 26,100 prompts drawn from 8 public instruction datasets (EN/JA general instructions, code, math, dialogue). Only the prompts were used. Every supervision target is Ornith-1.0-35B's own generation, plus the captured backbone hidden states.
Prompt sources (prompts only)
theblackcat102/evol-codealpaca-v1, m-a-p/CodeFeedback-Filtered-Instruction, openai/gsm8k, meta-math/MetaMathQA, OpenAssistant/oasst1, CohereLabs/aya_dataset, kunishou/hh-rlhf-49k-ja, kunishou/oasst1-89k-ja
Measured performance
Single-stream decode on 2× RTX 3090 (TP=2), vLLM nightly (0.23.1rc1), this exact checkpoint:
| Configuration | Decode tok/s | Acceptance rate | Per-position acceptance | Mean accepted length |
|---|
| No speculative decoding | ≈ 166 | — | — | 1.0 |
+ MTP, num_speculative_tokens=3 | ≈ 267 | 0.66 | 0.836 / 0.649 / 0.496 | 2.98 |
MTP row: 160 held-out mixed instruction + code prompts × 512 tokens each.
Baseline row: 28 coding prompts × 1024 tokens each.
k=3 gave the best single-stream throughput in testing. k=2 yields a higher per-draft acceptance rate (≈ 0.70–0.75) at slightly lower throughput.
MTP quantization
The MTP routed experts (mtp.layers.0.mlp.experts) are INT8-quantized with GPTQ, using the same AutoRound scheme as the backbone. Everything else in the MTP block stays in BF16.
How to use
vLLM is recommended (tested with vllm/vllm-openai:nightly, 0.23.1rc1).
To enable MTP speculative decoding:
1vllm serve <this-repo> \
2 --tensor-parallel-size 2 \
3 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
For text-only serving on 2×24 GB, --language-model-only --mamba-cache-mode align were used in testing.
If you're having quality issues while using MTP, it's likely a vLLM issue since MTP shouldn't degrade the output at all.
For more information, please see:
this reddit thread, and
this vLLM pullAcknowledgements
- Lorbus for the README.md format
- Sam McLeod for the Docker / 2× 3090 P2P configurations
- DeepReinforce AI for the base Ornith-1.0-35B model (post-trained on Qwen3.5-35B-A3B)
- Intel AutoRound team for the quantization framework
- vLLM project for the inference engine
- Qwen for the base MTP model
