Ornith-1.0-35B-oQ8e

Why this exists

• Peak memory usage drops substantially (see benchmarks below), making it practical on machines with less unified memory.
• Decode (token generation) throughput improves, since generation is memory-bandwidth bound and benefits from the smaller weight footprint.
• Quality is protected where it matters most: sensitive layers, MoE routers, shared experts, and vision/audio components are kept at higher precision instead of being blindly quantized (see below).
• The model keeps its full 35B-parameter (approximately 3B active) MoE architecture, tool-calling ability, and reasoning (<think>) behavior intact; only the numeric representation of the weights changes.

About the base model: Ornith-1.0-35B

• 35B total parameters, approximately 3B activated per token (MoE)
• 256 experts (8 active per token + 1 shared expert)
• Hybrid full-attention + linear (Gated DeltaNet) attention
• Vision encoder included (image-text-to-text / multimodal)
• Reasoning model: emits a <think>...</think> block before the final answer
• Strong tool-calling / OpenAI-compatible function-calling support
• State-of-the-art results among open models of comparable size on Terminal-Bench 2.1, SWE-bench Verified/Pro/Multilingual, and NL2Repo

Quantization method: oMLX Universal Dynamic Quantization

• Measures layer sensitivity using calibration prompts, comparing each quantized layer's output error against the float baseline.
• Allocates higher bits to sensitive layers and protected tensors via mixed precision, while keeping the overall bits-per-weight budget under a hard cap.
• Applies architecture-aware rules: MoE routers stay in fp16, shared experts and output-critical tensors are protected, vision/audio encoders are left unquantized, and SSM state tensors are preserved.

• Keeps the same oQ sensitivity plan, then layers an imatrix-weighted affine quantization step on top, the same idea llama.cpp's imatrix popularized: collect activation energy from representative prompts, then spend less quantization error on the input channels that matter most.
• Concretely: oQe runs calibration samples through the model and records average squared input activation, E[x^2], per quantized Linear input channel.
• For MoE SwitchLinear layers, it tracks one importance vector per expert, using router-selected expert indices, so rarely-activated experts still get proper coverage.
• Calibration sample count adapts upward from the requested amount when MoE expert coverage isn't yet sufficient.
• During affine quantization, scale/bias candidates are chosen to minimize sum(importance * (weight - dequantized_weight)^2) rather than plain unweighted MSE.
• If a tensor has no matching imatrix entry, it falls back to standard oQ for that tensor (unless strict coverage is enabled, in which case conversion fails instead of silently falling back).

Usage

Ornith is a reasoning model: the response will contain a <think>...</think> block before the final answer. Split on </think> if you only want the final answer.

Benchmarks

Single-request performance vs. context length

• Memory: oQ8e cuts peak memory by roughly 43 to 46% across all context lengths (e.g. 65.62 GB to 35.38 GB at 1K context; 70.63 GB to 40.18 GB at 200K context).
• Generation (TG) speed: oQ8e is about 31 to 37% faster at short-to-mid context (1K to 32K tokens), since decoding is memory-bandwidth bound and benefits directly from the smaller weights. The gap narrows around 64K context and becomes less consistent beyond that.
• Prompt processing (PP) speed: Mixed results, oQ8e is slightly faster at very short prompts but 6 to 26% slower at mid-to-large prompts (8K to 131K tokens), which is the expected trade-off of 8-bit compute (dequantization overhead during compute-bound prefill).
• Very long context (64K and above): Both TPOT and TG/PP throughput become noticeably less smooth for both models, expected behavior as unified memory pressure and cache effects start to dominate. This range is worth re-validating on your own hardware/config before relying on it.

Batching performance (fixed request size, varying concurrent batch)

• oQ8e scales much better with concurrency: at batch size 8 it reaches a 2.45x speedup over its own batch-1 throughput, versus only 1.13x for the BF16 base model. The extra memory headroom freed by 8-bit weights lets more requests be batched before hitting memory/bandwidth limits.
• End-to-end latency at batch 8 is roughly 1.8x lower with oQ8e (13.677 s vs 24.681 s), despite doing the same amount of work.
• The BASE model actually regresses at batch size 2 (0.64x, worse than single-request), while oQ8e scales up from batch 1 immediately. This suggests the base BF16 model is memory-constrained enough that low-batch concurrency isn't worthwhile on this hardware, whereas oQ8e has enough headroom to benefit from batching right away.

Limitations

• Quantization still introduces some numerical approximation vs. the original BF16 weights, even with oQe's sensitivity- and imatrix-aware protections. No formal accuracy/quality regression testing (e.g. on Terminal-Bench 2.1 or SWE-bench) has been run for this specific build; treat throughput/memory numbers above as validated, but re-check task quality for your own use case before production use.
• Any tensor without a matching imatrix calibration entry silently falls back to standard oQ precision for that tensor, which is expected but worth knowing if you inspect per-tensor bit allocation.
• Prompt-processing (prefill) throughput can be lower than the base model at large context sizes; this build is best suited to workloads dominated by generation rather than very long, one-off prompts.
• Behavior beyond approximately 64K context tokens is less predictable for both the base and quantized model and should be validated on your target hardware.

License

Citation