Qwen3.8-27B-ABLITERATED-mlx-q3km
MLX conversion of
Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16,
quantized with the
mixed_3_4 recipe as an equivalent to GGUF
Q3_K_M.
- 3.910 bits per weight, 13.4 GB across 3 safetensors shards
- Architecture
qwen3_5 (Qwen3_5ForConditionalGeneration), multimodal, hybrid
linear/full attention every 4th layer
- Requires mlx-vlm >= 0.6.13 (earlier versions have no
qwen3_5 module)
Why mixed_3_4 and not --q-bits 3
MLX's mixed-bit predicates are a deliberate port of llama.cpp's _K_M scheme,
not an approximation. Both give more bits to v_proj and down_proj in the
first eighth of layers, the last eighth, and every third layer between, plus a
high-bit lm_head. mixed_3_4 is 3-bit base with 4-bit on those sensitive
tensors, which is exactly Q3_K_M.
The measured result confirms it: 3.910 bpw here, against 3.91 bpw for the
official GGUF Q3_K_M (13.30 GB for ~27.2B params). A plain --q-bits 3 would
land near 3.5 bpw and would not match.
Note that mlx-vlm never quantizes the vision encoder at any level, so the
vision tower stays bf16.
Conversion
1mlx_vlm.convert \
2 --hf-path Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 \
3 --mlx-path ./Qwen3.8-27B-ABLITERATED-mlx-q3km \
4 --quantize --quant-predicate mixed_3_4
Usage
1pip install -U mlx-vlm
2
3# text
4mlx_vlm.generate --model wfiedler/Qwen3.8-27B-ABLITERATED-mlx-q3km \
5 --prompt "Explain unified memory in two sentences." --max-tokens 256
6
7# vision
8mlx_vlm.generate --model wfiedler/Qwen3.8-27B-ABLITERATED-mlx-q3km \
9 --prompt "Read this receipt and list every line item with its price." \
10 --image receipt.png --max-tokens 400
Quality vs the 5-bit conversion
Benchmarked against the q5 build (5.678 bpw, 18 GB) of the same base model, on
images with exact ground truth, 12 tasks x 3 trials at temperature 0.7.
| task | tests | q5 | this model |
|---|
| receipt | OCR, 14 fields | 14/14 x3 | 14/14 x3 |
| chart | bar values | 6/6 x3 | 6/6 x3 |
| table | 5x5 numeric cells | 8/8 x3 | 8/8 x3 |
| small text | small-font rare tokens | 6/6 x3 | 6/6 x3 |
| shapes (10) | count 3 groups | 6/6 x3 | 6/6 x3 |
| shapes (20) | count 4 scattered groups | 3/3 exact | 0/3 exact |
| photos | species ID | 3/3 | 3/3 |
| signs | real-world OCR | 2/3 + 1 loop | 3/3 |
| logic / recall | text reasoning | 3/3, 3/3 | 3/3, 3/3 |
On reading images the two are indistinguishable. Every OCR-shaped task ties
perfectly: all receipt fields, all chart values, all table cells, every rare
token in the small print.
The one real gap is counting many objects. Given 20 scattered shapes in four
colour/type groups, q5 answered correctly three times out of three; this model
answered 17, 19 and 21, never correct. The difference is visible in the output:
q5 enumerates shape by shape before totalling, while the 3-bit build emits a
terse number. The low-bit quant lost the habit of decomposing before answering.
Neither model does reliable mental arithmetic without a scratchpad; both compute
a four-item subtotal wrongly at temperature 0.7.
| q5 | this model |
|---|
| on disk | 18 GB | 12 GB |
| bits per weight | 5.678 | 3.910 |
| weights resident | 20.68 GB | 14.11 GB |
| peak during image generation | 23.24 GB | 16.67 GB |
| generation | 12.9 tok/s | 18.9 tok/s |
Measured on an Apple Silicon Mac with 64 GB unified memory, each model in its
own process. (GenerationResult.peak_memory is a process-global high-water mark
that does not reset between models, so comparing two in one process makes the
second inherit the first's peak.)
Use this build for OCR, document and chart work — 46% faster and 6.6 GB
lighter for no measurable loss. Use the 5-bit build when the task involves
counting or enumerating things in a scene.
Provenance and caveats
This is an abliterated model: the base has had refusal behaviour removed by its
authors. It will comply with requests a stock instruct model would decline.
Evaluate it before using it anywhere user-facing.
Quantization is lossy. The benchmark above covers a narrow slice of behaviour
and says nothing about long-context, tool-calling, or code quality at 3 bits.
All credit for the model itself goes to Blackfrost-AI and the Qwen team.
