This is a 4-bit GPTQ quantization of
empero-ai/Qwable-9B-Claude-Fable-5,
a full-parameter SFT distillation of Qwen3.5-9B on Claude Fable 5 and GPT-5.5 agentic-coding/reasoning
traces. All capabilities, usage notes, limitations, and licensing from the original model card apply
verbatim — please read it first: original README.
This is a community quantization by malvavisc0. It is not an
official release from Empero. All credit for the model itself goes to
Empero and the Qwen team.
200 (text backbone: attention + MLP, all 32 layers)
Avg / max GPTQ loss
9.4e-5 / 4.7e-4
What is (and isn't) quantized
GPTQ is applied only to the text backbone's linear layers. The following remain in bf16 (standard and
recommended — they are small and/or accuracy-critical):
the vision tower (Qwen3.5-Vision, ~0.6 GB) — the fine-tune was text-only anyway,
lm_head (lm_head=false),
embed_tokens (untied, 248,320 × 4096 — the largest single unquantized block),
norms / embeddings / non-linear params.
That is why an "Int4 9B" model is ~8.1 GB rather than ~4.5 GB: roughly half the bytes are unquantized
embedding tables. If you only need text inference, this is still a large VRAM win over the bf16 original
(~18 GB) and runs on a single 12 GB GPU with offloading.
Quantized module classes (per layer, all 32 layers covered):
⚠️ Calibration caveat. The base model is a coding/agentic/reasoning distill trained on traces up to
~74k tokens. This quantization was calibrated on general educational web text at 2048 tokens — it does
not match the model's specialty domain and does not exercise long context. Expect the largest
(still small, given the loss figures below) quality deltas on heavy coding tasks and very long inputs
compared to the bf16 original. A coding/agent-trace calibration set at longer seq_len would likely
improve this further.
Quantization loss (per-module MSE, from quant_log.csv)
value
Modules
200
Average loss
9.40e-5
Max loss
4.72e-4 (layer 28, linear_attn.in_proj_qkv)
Losses are uniformly tiny across all layers — no module tripped the 0.5% RTN fallback threshold. This
indicates a clean, well-conditioned quantization with no outlier layers.
How to use
The architecture is multimodal (Qwen3_5ForConditionalGeneration). Load it the same way as the original —
GPTQ weights are picked up automatically from quantization_config. You need
gptqmodel (or a recent transformers + optimum GPTQ backend)
and the Qwen3.5 kernels (flash-linear-attention + a CUDA-matched causal_conv1d), otherwise the
linear-attention layers fall back to slow PyTorch ops.
python
1import torch
2from transformers import AutoModelForImageTextToText, AutoTokenizer
34model_id ="malvavisc0/Qwable-9B-Claude-Fable-5-GPTQ-Int4"5tok = AutoTokenizer.from_pretrained(model_id)6model = AutoModelForImageTextToText.from_pretrained(7 model_id, dtype="bfloat16", device_map="auto",8)910messages =[{"role":"user","content":"Write a Python function that merges two sorted lists."}]11text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)12inputs = tok(text, return_tensors="pt").to(model.device)1314out = model.generate(15**inputs, max_new_tokens=2048, do_sample=True,16 temperature=0.7, top_p=0.95, top_k=20, repetition_penalty=1.05,17)18# Output starts with a <!thinking>...</thinking> reasoning block, then the final answer.19print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Sampling notes (inherited from the original card): the model is a reasoning model — every response
opens with a <!thinking> block; parse/strip it for end users. repetition_penalty≈1.05 prevents rare
non-terminating reasoning loops; allow generous max_new_tokens.
For vLLM / ExLlamaV serving, prefer their native GPTQ loader and a Marlin kernel for best throughput.
Known issues / caveats
Spurious top-level rope_parameters in config.json (rope_theta: 10000.0, plain default rope).
This field is not present in the original model and was injected during quantization; it conflicts with
the correct text_config.rope_parameters (rope_theta: 10000000, mrope with section [11, 11, 10]).
In practice the text sub-config takes precedence, but loaders that read the top-level field may emit warnings
or apply the wrong RoPE. If you see odd long-context behavior, delete the top-level rope_parameters from
config.json before loading.
quantization_config.meta.offload_to_disk_path contains a leftover local scratch path
(/tmp/gptqmodel_cfwi2by5) from the build host. It is metadata only and has no effect at load time.
Vision path is untested. The fine-tune was text-only; the vision tower is preserved unquantized but is
not validated. Treat this as a text model.
No quantitative benchmarks were run on this quantization. The GPTQ loss figures above are the only
quality signal; compare against the bf16 original on your own workload before relying on it.
apache-2.0, inherited from the base Qwen3.5-9B weights and the original fine-tune. The fine-tuning data
originates from generated traces of Claude Fable 5 and GPT-5.5 — see the
original card's Provenance & licensing section
for the third-party-terms caveat that applies to downstream commercial use.
