Ornith-1.0-35B-abliterix-UD-like-GGUF

Quantization Notes (UD-like / Unsloth-layout-reproduced)

Files

Note: The BF16 file is split into 2 shards because it exceeds Hugging Face's 50GB single-file limit (llama-gguf-split --split-max-size 40G); all other files are single files under 50GB. Shards must be kept in the same directory (llama.cpp auto-detects the -00001-of-00002 naming).

Quantization Approach Overview

• Reference source: Official UD-Q8_K_XL / UD-Q6_K_XL GGUFs and imatrix_unsloth.gguf_file published in unsloth/Ornith-1.0-35B-GGUF;
• Method: Parse the per-tensor storage types (per-tensor layout) from the official GGUFs, then apply the exact same layout to a BF16 GGUF converted from the fine-tuned (abliterix-ablated) model;
• Toolchain: llama.cpp convert_hf_to_gguf.py (BF16 conversion) + llama-quantize --imatrix --tensor-type (per-tensor override quantization following the official layout);
• Naming convention: UD-like / Unsloth-layout-reproduced — explicitly NOT official UD quantization (official UD requires Unsloth Studio's closed-source pipeline + proprietary calibration and cannot be publicly reproduced).

UD-like Per-Tensor Layout Details

UD-like-Q8_K_XL (reference: official UD-Q8_K_XL)

• Default tensor type: Q8_0
• The following 12 sensitive tensors are kept in BF16 (not quantized):
  • blk.34.ffn_down_exps.weight
  • blk.38.ffn_down_exps.weight
  • All of blk.39: attn_q/k/v/output, ffn_gate/up/down_exps, ffn_gate/up/down_shexp
• ssm_alpha / ssm_beta (30 each) kept in F32
• norm / router tensors (*_norm.weight, ffn_gate_inp*, ssm_a, ssm_conv1d, ssm_dt.bias) kept in F32 (architecture default)

UD-like_Q6_K_XL (reference: official UD-Q6_K_XL)

• Default tensor type: Q8_0
• ffn_gate_exps.weight / ffn_up_exps.weight of blk.0–blk.38 (78 MoE expert tensors in total) use Q6_K
• blk.39's ffn_gate/up_exps are the exception, kept in Q8_0
• ssm_alpha / ssm_beta kept in F32
• norm / router tensors kept in F32 (architecture default)

Reproducible Quantization Commands

Model Information

• Base model: ornith-ai/Ornith-1.0-35B (Qwen3.5-35B-A3B MoE fine-tune, 256 experts / 8 active, 40 layers, 262K context)
• Fine-tuning method: abliterix ablation (AbliteratedOrg/abliterix), vLLM backend (TP=2 + expert-parallel), EGA expert-granularity steering; LoRA merged back into BF16 weights
• Source repository: inkOrCloud/Ornith-1.0-35B-abliterix
• Reference layout: unsloth/Ornith-1.0-35B-GGUF

Disclaimer

• The UD-like models reproduce the official quantization layout (per-tensor storage types) only; they do not guarantee byte-level identity, task accuracy, or internal decision pipeline equivalence with official UD models;
• Official UD-Q8_K_XL / UD-Q6_K_XL are generated by Unsloth Studio (closed-source); this repository does not use its proprietary calibration process;
• Using the imatrix and layout reproduction inherits the official precision allocation for sensitive layers, but final quality should be verified empirically (we recommend comparing KLD / task metrics against the BF16 baseline).