GGUF quantizations of empero-ai/Qwythos-27B-v1 for llama.cpp, Ollama, LM Studio, jan, KoboldCpp, and other GGUF runtimes. Qwythos-27B-v1 is a dense, hybrid Qwen3.5-27B—the larger continuation of Qwythos-9B, trained on the same curriculum at a larger scale. Nothing was removed to make this conversion fit: its native MTP head, vision tower, and 1,048,576-token (1M) context capability are intact. This is the pre-RL v1 checkpoint, following SFT, DPO, and ESFT; an RL-trained v2 will follow, while v1 is a complete checkpoint in its own right.
For the full capability, training, and evaluation writeup, see the base model card. This card is intentionally focused on choosing a file and running it.
Files
The repository is named Qwythos-27B-v1-GGUF, but the files on disk deliberately use the shorter Qwythos-27B-* names below. Use those exact filenames when downloading or passing -m.
Normal text weights — trunk only (64 blocks)
File
Quant
Size
Notes
Qwythos-27B-Q4_K_M.gguf
Q4_K_M
15.78 GiB / 16.95 GB
recommended default
Qwythos-27B-Q5_K_M.gguf
Q5_K_M
18.11 GiB / 19.44 GB
K-quant
Qwythos-27B-Q6_K.gguf
Q6_K
20.91 GiB / 22.45 GB
K-quant
Qwythos-27B-Q8_0.gguf
Q8_0
26.63 GiB / 28.60 GB
8-bit quant
Qwythos-27B-BF16.gguf
BF16
50.11 GiB / 53.81 GB
full-precision conversion base
If you do not know which to pick, start with Q4_K_M.
MTP-enabled text weights (nextn_predict_layers = 1)
These include the native MTP head inside the GGUF. Use them with a recent llama.cpp build that supports MTP draft speculation, for example --spec-type draft-mtp.
File
Quant
Size
Notes
Qwythos-27B-MTP-Q4_K_M.gguf
Q4_K_M + MTP
16.05 GiB / 17.24 GB
recommended MTP default
Qwythos-27B-MTP-Q5_K_M.gguf
Q5_K_M + MTP
18.41 GiB / 19.76 GB
MTP K-quant
Qwythos-27B-MTP-Q6_K.gguf
Q6_K + MTP
21.25 GiB / 22.82 GB
MTP K-quant
Qwythos-27B-MTP-Q8_0.gguf
Q8_0 + MTP
27.05 GiB / 29.05 GB
MTP 8-bit quant
Qwythos-27B-MTP-BF16.gguf
BF16 + MTP
50.90 GiB / 54.66 GB
MTP, full-precision conversion base
Vision projector — for image input
File
Size
Notes
mmproj-Qwythos-27B-F16.gguf
0.86 GiB / 0.93 GB
vision encoder + projector at F16; required for images, pairs with any normal or MTP text quant
The vision tower is intact in this release. Download the mmproj alongside a text quant whenever you want image input.
What fits on a GPU?
These are practical weight-size-based estimates, not measured performance or VRAM benchmarks. They assume a modest context and leave room for runtime overhead and the KV cache; image input adds the 0.86 GiB mmproj. Longer contexts—especially the full 1M window—make the KV cache the dominant memory cost and may require CPU/KV offload or multiple GPUs regardless of weight quant.
Measured:Qwythos-27B-Q4_K_M.gguf fully offloaded to a single RTX 5090 (32 GB) at 4k context runs at ~74 tok/s generation, ~622 tok/s prompt (llama.cpp CUDA build, -ngl 99). One data point on one card, not a benchmark sweep — but it confirms a 27B at Q4_K_M is comfortably interactive on a single consumer GPU.
Quant
Approximate hardware guidance at modest context
Q4_K_M / MTP-Q4_K_M
The practical 24 GB-card starting point. Q4_K_M has the most headroom of these files for runtime and KV cache.
Q5_K_M / MTP-Q5_K_M
Can fit tightly on 24 GB at a modest context; 48 GB is the more comfortable single-GPU target.
Q6_K / MTP-Q6_K
A 48 GB card is the practical single-GPU target; on 24 GB, expect weight or KV offload.
Q8_0 / MTP-Q8_0
Plan on 48 GB for modest-context single-GPU use; 24 GB requires offload.
BF16 / MTP-BF16
The weights alone exceed 48 GB. Use an 80 GB-class GPU or multi-GPU/offload, with additional memory for runtime and KV cache.
Quick start
llama.cpp (llama-cli)
bash
1llama-cli \2 -m Qwythos-27B-Q4_K_M.gguf \3 -p "Explain the design tradeoffs in a Gated-DeltaNet hybrid model."\4 -n 16384\5 --temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05\6 -c 16384
ollama run hf.co/empero-ai/Qwythos-27B-v1-GGUF:Q4_K_M
LM Studio / jan / KoboldCpp
Download any text .gguf from this repository and load it in your runtime. Preserve the embedded Qwythos chat template if the runtime asks you to select or configure one.
Qwythos is a hybrid model: three Gated-DeltaNet linear-attention layers for every full-attention layer. Its Gated-DeltaNet state tensors are disproportionately sensitive to low-bit quantization, so these K-quants retain them at higher precision than the surrounding weights:
Tensor
Q4_K_M
Q5_K_M
Q6_K
ssm_alpha
Q8_0
Q8_0
Q8_0
ssm_beta
Q8_0
Q8_0
Q8_0
ssm_out
Q8_0
Q8_0
Q8_0
ssm_a, ssm_conv1d, ssm_dt, ssm_norm
F32
F32
F32
The entire Gated-DeltaNet state path is held at Q8_0 or better in every K-quant — no part of it is allowed to drop to Q4/Q5/Q6, regardless of the nominal quant level. This costs roughly 2–4% in file size over a flat K-quant and is the single most important difference between these files and a naive conversion of a hybrid model. Q8_0 and BF16 are uniform and need no overrides. MTP variants additionally pin nextn.eh_proj to Q8_0.
Built with llama.cpp pinned at commit d0f9d2e5, using architecture qwen35.
Identity and chat template
The embedded chat template identifies the model as: “You are Qwythos, a model created by Empero AI. Only bring up your identity if the user asks.” Use the runtime's chat/completions mode rather than hand-rolling a different prompt format where possible.
Function calling
The embedded chat template carries Qwen3.5's full tool-use spec, so tool calling works through llama-server's OpenAI-compatible endpoint without a wrapper — pass tools in your /v1/chat/completions request and parse the emitted <tool_call> blocks:
Given the terminal/agentic training behind this checkpoint, tool-augmented use is the intended deployment path.
Sampling recommendations
Qwythos is a reasoning model — responses open with a <think>…</think> block before the final answer. These match the shipped generation_config.json defaults:
Parameter
Recommendation
temperature
0.6 for agentic, harness, and tool-use work; 1.0 for open-ended reasoning or creative work
top_p
0.95
top_k
20
repeat_penalty
1.05
max_new_tokens
16,384 or more; reserve a generous budget for long reasoning traces and the final answer
Use repeat_penalty 1.05 rather than Qwen's default of 1.0 — it prevents rare non-terminating reasoning loops on long generations.
Long context (1M tokens)
The model configuration supports a 1,048,576-token window: a 4× YaRN extension over the 262,144-token native window. Set the runtime context to the amount you actually need—lower -c values substantially reduce KV-cache memory requirements.
The 1M setting is an inference-time capability, not a promise that a single GPU can hold a 1M KV cache. For roughly 512k context, a YaRN factor of 2.0 is preferable; if you will never exceed the native 262k window, remove scaling entirely. Static YaRN scaling can slightly degrade short-context quality. For the full 1M window, plan for multi-GPU or aggressive KV offload.
Verification
SHA256SUMS is included in this repository. Verify a download before loading it:
sha256sum -c SHA256SUMS
Limitations
Pre-RL v1. This release has completed SFT, DPO, and ESFT, but the planned RL-trained v2 is still to come.
Context costs memory. Weight size is only part of the hardware budget; long contexts need substantial KV-cache memory.
MTP is runtime-dependent. Use the trunk-only files when your GGUF runtime does not yet support draft-mtp speculation.
No standardized academic benchmark suite has been run for this checkpoint. Validate the model on the tasks and operating conditions that matter to your deployment.
Uncensored model. Add the application-level review and safety controls appropriate to an end-user deployment.
Stay in the loop
Sign up for the Empero newsletter at empero.org for releases, evals, and research notes.
Support / Donate
If this model helped you, consider supporting the project:
Qwythos-27B-v1 is derived from Qwen3.5-27B. The weights are released under Apache-2.0, inherited from the Qwen3.5-27B base, and are shared for research and experimentation as-is.
