Views
No views yet
Works with mainline llama.cpp. Thehy_v3architecture was merged upstream on 2026-07-14 (PR #25395) — any master build from that date (≥ b10005) loads these files directly, including--spec-type draft-mtpspeculative decoding via the bundled NextN/MTP layer.
| file | size | recipe |
|---|---|---|
Hy3-UD128-*.gguf | 116.7 GB total (sharded <45 GB) | routed experts: down IQ3_S · gate IQ2_S · up IQ3_XXS — attention Q5_K · shared expert + dense FFN Q6_K · output Q6_K · embeddings Q4_K |
Hy3-IQ4-UD-split-*.gguf | 143.9 GB total (sharded <45 GB) | max-quality edition for dual-device rigs: routed down IQ4_XS · gate/up IQ3_S · attention/actives Q8_0 |
Hy3.imatrix.gguf | small | importance matrix, ~125 chunks of 512 tokens, general-purpose calibration corpus |
ffn_down_exps at IQ3_S), the most quantization-sensitive expert tensor, funded by the gate projection at IQ2_S (a multiplicative mask, the most robust one); up sits in between at IQ3_XXS. Since routed experts are 97 % of the parameters, this is where the whole quality/size trade-off is decided.blk.80, ~2 GB) is never allocated at inference, so the resident footprint is ~107 GiB — it fits fully in the GPU/unified memory of 128 GB-class machines (Strix Halo / Apple Silicon / multi-GPU rigs), no CPU offload needed. Usable unified RAM on "128 GB" machines is really ~122–126 GB (firmware/OS reserve varies), and whatever else runs on the box counts against you. Comfortable defaults: 24K–32K with q8_0 KV, or 40K+ with q4_0 KV — on recent llama.cpp builds the Hadamard-rotated KV cache makes q4_0 nearly q8_0-quality, at half the size (~87 KB/token vs ~160). For longer context, offload a few expert layers to CPU/second GPU.--temp 0.9 --top-p 1.0. The chat template is embedded and resolved, so plain --jinja works for chat — for tool calling, add --chat-template-file hy3-chat-template.jinja (included in this repo, see Notes).1llama-server -m Hy3-UD128-00001-of-00003.gguf -ngl 99 -fa on --jinja \
2 -c 24576 -ctk q8_0 -ctv q8_0 --temp 0.9 --top-p 1.0 # or -c 40960 -ctk q4_0 -ctv q4_01llama-server -m Hy3-IQ4-UD-split-00001-of-00004.gguf --device CUDA0,ROCm0 -ngl 99 -fa on --jinja \
2 -c 16384 -ctk q4_0 -ctv q4_0 \
3 -ot "ffn_.*_exps=ROCm0" -ot "output=CUDA0" -ot "token_embd=ROCm0" \
4 --temp 0.9 --top-p 1.0hy3-chat-template.jinja (--jinja --chat-template-file hy3-chat-template.jinja). The embedded template renders the eos token as literal text after assistant turns; llama.cpp's tool-call autoparser bakes that literal into its induced grammar, and at generation time the real EOG token duplicates it — the grammar parse fails mid-stream (common_chat_peg_parse: unparsed peg-native output → 500 on streaming tool calls) and the eos string leaks into plain-chat content. The repo template is identical except it renders eos_token only to terminate past assistant turns in the history, never after the final (currently-generating) turn — so the induced grammar never carries a trailing eos literal, while multi-turn history stays correctly delimited. Stopping is handled by GGUF eos_token_id metadata, unaffected. Embedded metadata will be corrected in the next file revision.blk.80) is included and measured working with llama.cpp PR #25395's --spec-type draft-mtp: on the same Strix Halo, decode goes from ~17 t/s to 24.3 t/s on code (+40%) and ~20 t/s on prose, with 91% draft acceptance at temp 0.9 (--spec-draft-p-min 0.75 is required — the MTP head is single-depth-trained and the p_min=0 default makes speculation a net slowdown). Multi-turn prompt-cache reuse is unaffected.general.architecture = hy_v3, matching the merged upstream arch — no rename needed, mainline loads them as-is. (History: the port originally shipped as hy-v3; when #25395 merged under hy_v3, shard 1 of each quant was re-uploaded with a metadata-only patch — tensor data is byte-identical. If you kept an old download, either re-fetch shard 1 or run the included gguf-arch-rename.py.)blk.80) has no imatrix data (it is never activated), so any very-low-bit type that requires an importance matrix — iq3_xxs, iq2_*, iq1_* — will abort on blk.80.ffn_down_exps at the end of the run. Pin that layer to an imatrix-exempt type first: --tensor-type 'blk\.80\.=q4_0' (its precision is irrelevant since it is skipped at inference). k-quants, iq3_s, and iq4_xs are unaffected.transformers verified at f32 (KL = 0.0, exact top-1 match on English/code/French/chat prompts); tokenizer verified against AutoTokenizer.