Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4-MTP-GGUF
Experimental GGUF conversion of AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4 with an embedded native Qwen3.6 MTP block.
This is a self-contained NVFP4 GGUF with embedded native MTP tensors.
It does not require an external MTP-only draft model, but this embedded-MTP path is experimental.
Status
Experimental.
The model loads and performs native MTP speculative decoding in llama.cpp. However, on the tested system, using the same MTP tensors through a standalone MTP-only GGUF via --model-draft was slightly faster than embedding the tensors directly into the GGUF.
For most users, the standard NVFP4 GGUF plus external --model-draft is recommended.
Model Details
- Source model:
AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4
- Base model:
deepreinforce-ai/Ornith-1.0-35B
- Format: GGUF
- Quantization: NVIDIA NVFP4
- Architecture: Qwen3.6 Mixture-of-Experts with hybrid attention
- MTP: Embedded native Qwen3.6 MTP block
- Parameters: 35B total, approximately 3B activated per token
- Purpose: Local inference and native MTP speculative decoding with llama.cpp
Embedded MTP
This model was created by grafting the native Qwen3.6 MTP block into the converted NVFP4 GGUF.
The resulting GGUF contains:
1qwen35moe.block_count = 41
2qwen35moe.nextn_predict_layers = 1
Additional tensors:
Only the native MTP block was added. The original transformer layers blk.0.* through blk.39.* were not modified.
Recommended graft source:
Compatibility
A recent version of llama.cpp with Qwen3.6 MoE, NVFP4, and native MTP support is required.
Tested with:
- Windows
- NVIDIA GeForce RTX 5070 Ti 16 GB
- NVIDIA GeForce RTX 5060 Ti 16 GB
- llama.cpp CUDA backend
- Embedded native Qwen3.6 MTP speculative decoding
Older llama.cpp builds may fail to recognize the nvfp4 tensor type, may not correctly load associated scale tensors, or may lack compatible Qwen3.6 MoE/MTP support.
Performance may vary with llama.cpp build, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.
Usage
llama-server
1llama-server \
2 -m Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4-MTP.gguf \
3 --spec-type draft-mtp \
4 --spec-draft-n-max 3 \
5 --flash-attn on \
6 -ngl 99
llama-cli
1llama-cli \
2 -m Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4-MTP.gguf \
3 --spec-type draft-mtp \
4 --spec-draft-n-max 3 \
5 --flash-attn on \
6 -ngl 99
Multi-GPU example
1llama-server \
2 -m Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4-MTP.gguf \
3 --spec-type draft-mtp \
4 --spec-draft-n-max 3 \
5 --split-mode layer \
6 --tensor-split 1,1 \
7 --flash-attn on \
8 -ngl 99
The best tensor split depends on available VRAM, GPU speed, PCIe topology, context length, and KV-cache placement. An even split is only a starting point.
Runtime Recommendation
Although the embedded MTP GGUF functions correctly, local benchmarks showed that using the standalone MTP-only GGUF via --model-draft was slightly faster than embedding the same tensors directly into the GGUF.
Recommended for most users:
1llama-server \
2 -m Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4.gguf \
3 --model-draft Qwen3.6-35B-A3B-MTP-ONLY-Q6_K.gguf \
4 --spec-type draft-mtp \
5 --spec-draft-n-max 3
Use this embedded-MTP GGUF mainly if you want a self-contained file or want to experiment with native MTP grafting.
Benchmark Summary
Local mixed-task benchmark:
| Mode | Aggregate acceptance | Wall time | Notes |
|---|
| No MTP | n/a | 21.88 s | Baseline (standard decoding) |
External MTP-only (--model-draft, Q6_K) | 95.09% | 22.33 s | Recommended; highest acceptance and best speculative decoding performance in local tests |
| Embedded MTP | 93.17% | 24.51 s | Functional, but slower than the external draft model in local tests |
The embedded model works correctly, but --model-draft currently appears to be the better runtime path on the tested setup.
Results may vary depending on llama.cpp version, CUDA graph behavior, GPU offload, batch/ubatch settings, context length, quantization, and prompts.
Suggested MTP Settings
A reasonable general-purpose starting point is:
For translation, role-play, creative writing, conversational output, or open-ended explanations, start with:
For JSON, fixed templates, repeated patterns, and deterministic code completion, try:
Grafting Method
The embedded MTP block was generated by grafting the standalone MTP-only GGUF.
This demonstrates that MTP-only GGUF files can serve two purposes:
- Standalone draft model via
--model-draft
- Graft source for creating a self-contained GGUF
The graft process updates:
1qwen35moe.block_count: 40 -> 41
2qwen35moe.nextn_predict_layers: 1
and appends:
It does not modify:
1tokenizer
2vocabulary
3chat template
4RoPE settings
5original transformer layers
Verification
Check the embedded MTP metadata and tensors:
1python .\gguf-py\gguf\scripts\gguf_dump.py Qwen3.6-35B-A3B-NVFP4-MTP.gguf |
2 Select-String "block_count|nextn_predict_layers|blk\.40"
Expected output should include:
1qwen35moe.block_count = 41
2qwen35moe.nextn_predict_layers = 1
3blk.40.attn_k.weight
4blk.40.nextn.eh_proj.weight
5blk.40.nextn.shared_head_norm.weight
Notes
- This GGUF preserves NVIDIA's NVFP4 tensor type; it is not equivalent to
Q4_K_M, Q4_K_S, or an Unsloth dynamic quant.
- The embedded MTP block is experimental and may not be faster than using
--model-draft.
- The model is a 35B-total-parameter MoE with approximately 3B parameters activated per token; runtime memory requirements still depend on the full stored checkpoint rather than only the active parameter count.
- NVIDIA's source checkpoint was prepared for ModelOpt and vLLM. llama.cpp support is a separate community implementation and may behave differently from NVIDIA's reference runtime.
- Native MTP accelerates token generation but does not improve prompt-prefill speed in the same way.
- Very short benchmark outputs are more sensitive to run-to-run variance.
- Results are specific to the tested hardware, llama.cpp build, prompts, runtime options, and context configuration.
- Although standard decoding completed slightly faster on this short benchmark, native MTP enables speculative decoding and substantially increases token generation throughput (up to ~115 tok/s in these tests). Acceptance rate is therefore a more meaningful metric than total wall time when evaluating MTP effectiveness.
Related Projects
- Standard NVFP4 GGUF:
AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4
- Grafting utility:
gguf-graft-mtp
Credits
- Qwen Team / Alibaba Cloud — Qwen3.6-35B-A3B
- deepreinforce-ai / Ornith-1.0-35B — Qwen3.6-35B-A3B
- NVIDIA — ModelOpt and the original NVFP4 checkpoint
- ggml-org — llama.cpp, GGUF, NVFP4 inference support, and native MTP support
- a4lg — MTP-only GGUF subset used as a graft source
License
The source model is distributed under the Apache License 2.0.
Users should review the upstream AEON-7/Ornith-1.0-35B-AEON-Ultimate-Uncensored-NVFP4, deepreinforce-ai/Ornith-1.0-35B, and the MTP-only GGUF source model cards before redistribution or commercial use.
