Ornith 1.0 35B AEON Ultimate Uncensored - GGUF
GGUF quantizations of
AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-BF16, produced from the
BF16 source weights using importance-matrix calibration.
Two variants are provided: standard trunk (no MTP) and MTP-grafted (with Multi-Token Prediction block for speculative decoding).
Files
Standard Trunk (no MTP)
For standard autoregressive inference. Smaller files, no speculative decoding overhead.
| File | Quant | Size | BPW | Target GPU |
|---|
ornith-aeon-35b-Q8_0.gguf | Q8_0 | 35 GB | ~8.5 | 48GB+ (near-lossless) |
ornith-aeon-35b-Q6_K.gguf | Q6_K | 27 GB | ~6.6 | 32GB+ (high quality) |
ornith-aeon-35b-Q5_K_M.gguf | Q5_K_M | 24 GB | ~5.7 | 24GB (quality-first) |
ornith-aeon-35b-Q4_K_M.gguf | Q4_K_M | 21.2 GB | ~4.8 | 24GB (balanced) |
ornith-aeon-35b-Q4_K_S.gguf | Q4_K_S | 19.9 GB | ~4.6 | 24GB fallback |
ornith-aeon-35b-IQ4_XS.gguf | IQ4_XS | 18.7 GB | ~4.3 | 20GB (RTX 4000 Ada) |
ornith-aeon-35b-Q3_K_M.gguf | Q3_K_M | 16.8 GB | ~3.9 | 16GB GPUs |
ornith-aeon-35b-Q2_K.gguf | Q2_K | 12.9 GB | ~3.0 | 12GB GPUs |
ornith-aeon-35b-IQ1_M.gguf | IQ1_M | 8.2 GB | ~1.8 | 8GB GPUs |
ornith-aeon-35b.imatrix | - | 184 MB | - | Importance matrix |
MTP-Grafted (with Multi-Token Prediction)
These GGUFs contain all
785 MTP tensors grafted from the base
Qwen/Qwen3.5-35B-A3B model, adding a full MTP prediction block (blk.40) with 256 MoE experts. Use with
--spec-type draft-mtp for speculative decoding. MTP is bundled in the GGUF -- no separate draft file needed.
| File | Quant | Size | BPW | Target GPU |
|---|
ornith-aeon-35b-MTP-Q8_0.gguf | Q8_0 | 36 GB | ~8.5 | 48GB+ (near-lossless) |
ornith-aeon-35b-MTP-Q6_K.gguf | Q6_K | 28 GB | ~6.6 | 32GB+ (high quality) |
ornith-aeon-35b-MTP-Q5_K_M.gguf | Q5_K_M | 24 GB | ~5.7 | 24GB (quality-first) |
ornith-aeon-35b-MTP-Q4_K_M.gguf | Q4_K_M | 21.7 GB | ~4.8 | 24GB (balanced) |
ornith-aeon-35b-MTP-Q4_K_S.gguf | Q4_K_S | 20.4 GB | ~4.6 | 24GB fallback |
ornith-aeon-35b-MTP-IQ4_XS.gguf | IQ4_XS | 19.2 GB | ~4.3 | 20-24GB |
ornith-aeon-35b-MTP-Q3_K_M.gguf | Q3_K_M | 17.2 GB | ~3.9 | 16GB GPUs |
ornith-aeon-35b-MTP-Q2_K.gguf | Q2_K | 13.3 GB | ~3.0 | 12GB GPUs |
ornith-aeon-35b-MTP.imatrix | - | 184 MB | - | Importance matrix |
Why Two Variants?
The original AEON fine-tune lost its 785 MTP weight tensors. HuggingFace Transformers' AutoModelForCausalLM silently drops all mtp.* tensors during loading (_keys_to_ignore_on_load_unexpected = [r"^mtp.*"]). The mtp_num_hidden_layers: 1 in config.json is orphaned metadata from the base Qwen3.5-35B-A3B model.
The MTP-grafted variants restore all 785 MTP tensors (~488 MB BF16) by copying them from the base Qwen3.5-35B-A3B model. This works because Ornith shares identical architecture, hidden dimensions, expert count, and embedding space with its base model. Community-measured acceptance rates for grafted MTP: 58-100%.
MTP Benchmark Results
Tested: MTP-Q4_K_M on DGX Spark (GB10)
| Mode | Prompt | Generation |
|---|
| Standard (no MTP) | 47.4 t/s | 34.9 t/s |
Embedded --spec-type draft-mtp | 46.2 t/s | 62.8 t/s |
1.8x generation speedup with the bundled MTP head on a single GPU.
Embedded MTP vs separate draft model
An important distinction for MoE speculative decoding:
- Embedded MTP head (bundled in the GGUF,
--spec-type draft-mtp): Net positive. The MTP head shares the model's KV cache and embedding space. Measured +27-80% generation speedup depending on workload and hardware.
- Separate draft model (
-md with a standalone GGUF): Net negative for MoE. A separate draft model triggers expert-union overhead during batch verification -- more expert weight blocks must be read from VRAM, exceeding the savings. Benchmarks show -18% to -52% regression.
The MTP-grafted GGUFs in this repo use the embedded approach.
How to Run
Standard inference (recommended for most users)
1llama-server \
2 -m ornith-aeon-35b-Q4_K_M.gguf \
3 -a ornith-aeon-35b \
4 --host 0.0.0.0 --port 8083 \
5 -ngl 99 \
6 --n-cpu-moe 3 \
7 -fa on \
8 -ctk q4_0 -ctv q4_0 \
9 -c 32768 \
10 -b 2048 -ub 768 \
11 -np 1 -cb \
12 -n 16384 \
13 --temp 0.6 --top-k 20 --top-p 0.95 \
14 --repeat-penalty 1.1 \
15 --jinja \
16 --reasoning-format deepseek \
17 --reasoning-budget 1024
With MTP speculative decoding (faster generation)
Requires llama.cpp built from latest master (b9606+). MTP is bundled in the GGUF -- no separate draft file needed.
1llama-server \
2 -m ornith-aeon-35b-MTP-Q4_K_M.gguf \
3 -a ornith-aeon-35b \
4 --spec-type draft-mtp \
5 --host 0.0.0.0 --port 8083 \
6 -ngl 99 \
7 --n-cpu-moe 3 \
8 -fa on \
9 -ctk q4_0 -ctv q4_0 \
10 -c 32768 \
11 -b 2048 -ub 768 \
12 --parallel 1 \
13 --temp 0.6 --top-k 20 --top-p 0.95 \
14 --repeat-penalty 1.1 \
15 --jinja \
16 --reasoning-format deepseek \
17 --reasoning-budget 1024
RTX 4000 Ada 20GB
1llama-server \
2 -m ornith-aeon-35b-MTP-IQ4_XS.gguf \
3 -a ornith-aeon-35b \
4 --spec-type draft-mtp \
5 --host 0.0.0.0 --port 8083 \
6 -ngl 99 \
7 --n-cpu-moe 5 \
8 -fa on \
9 -ctk q4_0 -ctv q4_0 \
10 -c 8192 \
11 -b 1024 -ub 512 \
12 --parallel 1 \
13 --temp 0.6 --top-k 20 --top-p 0.95 \
14 --repeat-penalty 1.1 \
15 --jinja \
16 --reasoning-format deepseek \
17 --reasoning-budget 1024
High quality on 32GB+ GPU
1llama-server \
2 -m ornith-aeon-35b-MTP-Q6_K.gguf \
3 -a ornith-aeon-35b \
4 --spec-type draft-mtp \
5 --host 0.0.0.0 --port 8083 \
6 -ngl 99 \
7 -fa on \
8 -ctk q8_0 -ctv q8_0 \
9 -c 32768 \
10 -b 2048 -ub 768 \
11 --parallel 1 \
12 --temp 0.6 --top-k 20 --top-p 0.95 \
13 --repeat-penalty 1.1 \
14 --jinja \
15 --reasoning-format deepseek \
16 --reasoning-budget 1024
Speculative Decoding Options
All available in llama.cpp latest master. Work on any CUDA GPU (Ada Lovelace, Ampere, Blackwell):
| Strategy | Flag | Draft Model? | Notes |
|---|
| MTP (embedded) | --spec-type draft-mtp | Bundled in MTP GGUFs | Requires --parallel 1. Measured +80% on Spark. |
| Eagle3 | --spec-type draft-eagle3 | Separate GGUF | PR #18039. No pre-trained drafter for this model yet. |
| DFlash | --spec-type draft-dflash | Separate GGUF | z-lab/Qwen3.5-35B-A3B-DFlash. Best results on vLLM/SGLang at BF16/FP8. |
| N-gram | --spec-default | None | Zero VRAM cost. Marginal benefit on structured output. |
Key Parameters
| Parameter | Value | Why |
|---|
-ngl 99 | Offload all layers to GPU | Maximizes speed |
--spec-type draft-mtp | MTP speculative decoding | Only with MTP-grafted GGUFs |
--parallel 1 | Single slot | Required for MTP |
--n-cpu-moe N | Offload N MoE layers to CPU | Saves ~0.8GB per layer |
-fa on | Flash attention | Reduces KV cache memory |
-ctk q4_0 -ctv q4_0 | Quantize KV cache | Saves ~60% KV cache VRAM |
--jinja | Jinja2 chat template | Required for this model |
--reasoning-format deepseek | Thinking mode | Model uses think tags |
If It OOMs
- Increase
--n-cpu-moe (trades speed for VRAM)
- Lower context:
-c 8192 or -c 4096
- Lower batch:
-b 512 -ub 256
- Use a trunk (non-MTP) file to save ~0.5-1GB
- Use a smaller quant
Model Details
- Architecture: Qwen3.5-MoE (Mixture of Experts)
- Total Parameters: ~35B
- Active Parameters per Token: ~9B (4 of 64 experts active)
- MTP Tensors: 785 (grafted from base model in MTP variants)
- Chat Template: ChatML
- Thinking/Reasoning: DeepSeek-style think tags
- Context: Up to 131K tokens (limited by available VRAM)
Quantization Details
- Source: BF16 safetensors from AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-BF16
- MTP Graft Source: 785 mtp.* tensors from Qwen/Qwen3.5-35B-A3B (~488 MB BF16)
- Importance Matrix: Calibrated on diverse coding, debugging, system design, and reasoning prompts
- llama.cpp: Built from latest master (post-b9606, with MTP/Eagle3/DFlash support merged)
- Platform: DGX Spark (aarch64, CUDA 13.0)
Credits
- Base model by AEON-7
- MTP weights from Qwen/Qwen3.5-35B-A3B
- Quantization by vcruz305
