The source is a ~35B-total Mixture-of-Experts model (qwen3_5_moe, 256 experts / 8 active). At 4.0 bpw the weights occupy roughly 18–20 GB, which fits a single 24 GB GPU (e.g. RTX 3090/4090) with room for context — unlike higher bitrates which exceed 24 GB.

Details

Format: EXL3 (exllamav3), measurement + quantization in one pass with --hq.
Tool: exllamav3 v0.0.43, PyTorch 2.8.0 + CUDA 12.8, on an RTX 3090.
Embeddings / norms: kept at FP16; lm_head at ~6 bpw.
Vision tower: the model is multimodal (image+video); the vision encoder is kept at original precision and copied through. (exllamav3 v0.0.43 required a one-line patch to whitelist the qwen3_5_moe_vision vision config model_type.)
MTP head excluded: the base config declares mtp_num_hidden_layers = 1, but the repository ships no mtp.* weights, so the optional multi-token-prediction (speculative-decoding draft) head is omitted. This does not affect the main model.

Usage

Load with exllamav3, or serve via TabbyAPI.

python

from exllamav3 import Model, Config, Cache, Tokenizer
model = Model.from_config(Config.from_directory("/path/to/Ornith-1.0-35B-EXL3-4.0bpw"))

Calibration

Quantized using exllamav3's built-in calibration dataset (a broad text mix), default rows/length.

hakuturu583/Ornith-1.0-35B-EXL3-4.0bpw

Ornith-1.0-35B-EXL3-4.0bpw

Why 4.0 bpw?

Details

Usage

Calibration

hakuturu583/
Ornith-1.0-35B-EXL3-4.0bpw