Qwen3-8B EAGLE3 Speculator GGUF
This repository contains a GGUF conversion of RedHatAI/Qwen3-8B-speculator.eagle3 for use with llama.cpp EAGLE3 speculative decoding.
[!IMPORTANT]
This is not a standalone chat model. It is an EAGLE3 draft/speculator model and must be used together with the matching target/verifier model.
- Target model:
Qwen/Qwen3-8B
- Speculator source:
RedHatAI/Qwen3-8B-speculator.eagle3
- Runtime: llama.cpp with
--spec-type draft-eagle3
Files
| File | Type | Notes |
|---|
Qwen3-8B-speculator.eagle3-F16.gguf | EAGLE3 speculator GGUF | Converted from the original RedHatAI safetensors checkpoint |
Usage with llama.cpp
Example:
1llama-server \
2 -m Qwen3-8B-Q4_K_M.gguf \
3 -md Qwen3-8B-speculator.eagle3-F16.gguf \
4 --spec-type draft-eagle3 \
5 --spec-draft-n-max 4 \
6 --spec-draft-p-min 0.5 \
7 -c 32768 \
8 -ngl 99 \
9 -fa on
Windows CMD example:
1llama-server.exe ^
2 -m Qwen3-8B-Q4_K_M.gguf ^
3 -md Qwen3-8B-speculator.eagle3-F16.gguf ^
4 --spec-type draft-eagle3 ^
5 --spec-draft-n-max 4 ^
6 --spec-draft-p-min 0.5 ^
7 -c 32768 ^
8 -ngl 99 ^
9 -fa on
Important Notes
This GGUF file is only the draft/speculator model. You still need a compatible GGUF of the target model, such as a GGUF conversion or quantization of Qwen/Qwen3-8B.
Do not use this speculator with unrelated models such as Qwen3-14B, Qwen3-32B, Gemma, Llama, or Qwen3.6 models. EAGLE3 speculators are target-specific.
Tested Configuration
Tested with:
- Runtime: llama.cpp with EAGLE3 support
- Target model: Qwen3-8B GGUF
- Draft model: this EAGLE3 GGUF
- Example settings:
--spec-type draft-eagle3--spec-draft-n-max 4--spec-draft-p-min 0.5
Local benchmark observations may vary depending on GPU, quantization, context length, batch size, and prompt type.
Benchmark Notes
In local testing, EAGLE3 showed stronger gains on structured or predictable outputs such as:
- code generation
- JSON output
- repeated pattern generation
- short factual responses
It was less effective on some tasks such as translation, creative writing, and long reasoning, where draft acceptance may be lower.
Conversion
Converted with llama.cpp convert_hf_to_gguf.py using the original speculator repository and the matching target model directory.
Example conversion command:
1python convert_hf_to_gguf.py \
2 RedHatAI/Qwen3-8B-speculator.eagle3 \
3 --outtype f16 \
4 --target-model-dir Qwen/Qwen3-8B \
5 --outfile Qwen3-8B-speculator.eagle3-F16.gguf
PowerShell example:
1python .\convert_hf_to_gguf.py `
2 "E:\OLLAMA_MODELS\Qwen3-8B-speculator.eagle3" `
3 --outtype f16 `
4 --target-model-dir "E:\OLLAMA_MODELS\Qwen3-8B" `
5 --outfile "E:\OLLAMA_MODELS\Qwen3-8B-speculator.eagle3-F16.gguf"
Credits
Original EAGLE3 speculator model by RedHatAI:
RedHatAI/Qwen3-8B-speculator.eagle3
Target model:
GGUF support and runtime:
License
This repository is a converted GGUF version of the original speculator model. The original model license and usage terms apply. Please refer to the upstream repositories for full license details.
