EXL3 quantization of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 (commit 9d85770e5eb602322b4bceef55beda357e0bd0ca), an abliterated Qwen3.8-27B multimodal model.

Third-party derivative, unevaluated by upstream. This is a community EXL3 build, not an official Qwen or Blackfrost release. Blackfrost's own model card states: "Any additional direction editing, fine-tuning, merging, pruning, quantization, or other weight change creates an artifact Blackfrost has not evaluated unless a new report explicitly states otherwise." That applies directly here — quality, refusal behavior, reasoning, tool use, vision/video behavior, and long-context stability of this specific 2.25 bpw quant have not been assessed by Blackfrost and may differ from the BF16 reference. This is a weight-level research checkpoint with a deliberately reduced refusal surface; it is not a safety-stock model and must not be represented, deployed, or evaluated as one.

Architecture: Qwen3_5ForConditionalGeneration (hybrid linear + full attention, 64 layers, MTP)
Parameters: ~27.78 B (text + vision)
Native context: 262 144 tokens
Vocab: 248 320 (Qwen3.8)
Build wall time: 1 h 33 min 14 s end-to-end (per-module sum: 1 h 04 min) on a single RTX 4090 (24 GB), exllamav3 v1.4.2, calibration 250×2048, codebook mul1, out_scales always. The per-module sum is the cumulative time the model modules spent actively being processed (sum of per-module [NN.NN s] markers across the log).
VRAM (observed, TabbyAPI, single user): ~15.6 GB (16 013 MiB) with cache_size: 65536, cache_mode: Q8, max_seq_len: 65536 — leaves ~8.4 GB headroom on a 24 GB card.
Quantization: EXL3 v1.4.2
Disk size: 10.74 GB (2 shards: 7.98 + 2.77 GB)

Quantization

Component	bpw	Notes
Transformer body (64 layers)	2.25 (target average; real per-layer range 2.02–2.33, weighted average 2.27)	`-b 2.25` — exllamav3 v1.4.2 distributes bits heterogeneously across three groups: 10 layers at 2.02 bpw (the deepest linear-attention subset, layers 26/28-30/32-34/36-38), 16 layers at 2.29 bpw (positions 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47, 51, 55, 59, 63 — every 4th starting at index 3), 38 layers at 2.33 bpw (the rest)
`embed_tokens`	16	not quantized (exllamav3 stores embedding tables raw bf16, not via trellis)
`lm_head`	6	`-hb 6` — protects output quality
MTP (1 layer)	4	`-mb 4` (exllamav3 default)
Vision tower (ViT)	16	`-vb 16` — effectively unquantised
Norm tensors	16	unquantised

Codebook: mul1 (default)
Out scales: always
Calibration: exllamav3 v1.4.2 standard data (c4/code/multilingual/technical/wiki/tiny), 250 rows × 2048 cols
Per-module SQNR (65 of 70 modules; 5 stored unquantized at 16 bpw and excluded because exllamav3 prints sqnr=0 for them): median 25.37 dB, mean 27.24 dB, min 21.88 dB (layer 51, bpw=2.29), max 43.00 dB (lm_head, bpw=6.00). Layer SQNR drops with depth: layers 0–2 land at 32–36 dB, layers 3–19 at 26–33 dB, layers 20+ at 22–26 dB — deeper layers are the SQNR floor.

Effective average over the whole file (including head/MTP/vision): 3.09 bpw (10.74 GB × 8 / 27.78 B params).

Files

Sizes are in bytes. SHA-256 is the first 16 hex chars (truncated).

File	Size (B)	SHA-256	Role
`.gitattributes`	1,570	`D8EBD820E12F314D`	git LFS filter rules (synced from upstream)
`chat_template.jinja`	10,446	`6E0F10E53C40D7F3`	Jinja template (modified — see below)
`config.json`	4,622	`0B8AA00D1C109719`	architecture + embedded `quantization_config.bits=2.25`
`generation_config.json`	202	`E70C136C1B78DDC1`	eos/bos + sampling defaults
`LICENSE`	11,343	`50CBAB8A892C5F29`	Apache License 2.0
`model.safetensors.index.json`	238,664	`8EA407462021FC26`	tensor → shard map (offsets reflect actual shard sizes)
`model-00001-of-00002.safetensors`	8,567,230,103	`D54F25C231F60DB7`	weights shard 1 (7.98 GiB)
`model-00002-of-00002.safetensors`	2,969,434,880	`F3BCAF094FB7A2E4`	weights shard 2 (2.77 GiB)
`preprocessor_config.json`	391	`5102CC0567B75A34`	vision/image preprocessor (Qwen2VL)
`quantization_config.json`	629,026	`5E148054E3FE6C9A`	exllamav3 per-tensor bit/storage map
`tokenizer.json`	12,809,320	`0997F410C57A1F4E`	HF fast tokenizer (BPE)
`tokenizer_config.json`	19,349	`3E63E525D8309DCE`	tokenizer metadata + embedded chat template
`video_preprocessor_config.json`	386	`00BD47A5EAAF8760`	video preprocessor (Qwen3VL)

Hashes for the 8 infrastructure files are byte-identical to the corresponding files in the Blackfrost BF16 source (synced via the 3.75bpw/2.75bpw quant builds). The two quant-meta files (config.json, quantization_config.json) and the shard files are necessarily different per bpw; their hashes here are the actual ones for this build.

Modifications vs upstream

This 2.25 bpw build contains the following file categories:

Byte-identical to the Blackfrost BF16 source (8 files): .gitattributes, LICENSE, chat_template.jinja (upstream variant, before the developer-role patch), generation_config.json, tokenizer.json, tokenizer_config.json, preprocessor_config.json, video_preprocessor_config.json.
exllamav3-generated quant metadata (2 files):
- config.json — the upstream JSON is preserved unchanged; exllamav3 appended a quantization_config block with bits: 2.25, head_bits: 6, ....
- quantization_config.json — exllamav3 per-tensor bit/storage map, not present in the BF16 source.
2.25 bpw build artifacts (3 files): the two weight shards model-0000{1,2}-of-00002.safetensors and the model.safetensors.index.json (whose tensor→shard offsets reflect the actual 2.25 shard sizes, not the BF16 ones).

One further intentional patch on top of the upstream files:

chat_template.jinja — added or message.role == "developer" in three places so that the template also accepts role=developer as an alias for role=system. The upstream template only recognises system/user/assistant/tool; the developer role is used by some OpenAI-style APIs for the higher-priority instruction.

Quick start (TabbyAPI + exllamav3)

Full tabby-config-2.25.yml used to serve this model on a 24 GB GPU (RTX 4090 verified):

yaml

network:
  host: 0.0.0.0
  port: 5000
  disable_auth: true       # only safe behind localhost / a trusted network

model:
  model_dir: /app/models
  model_name: Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw

  # --- KV cache (24 GB budget) ---
  cache_size: 65536        # 64k tokens (use 32k for tighter headroom at 2.25 bpw)
  cache_mode: Q8           # 8-bit cache; halves VRAM vs FP16, < 10% speed cost
  max_seq_len: 65536       # cap request length to the cache

  # --- EXL3-specific (optional) ---
  chunk_size: 2048
  output_chunking: true
  inline_model_loading: true

  gpu_split_auto: true
  autosplit_reserve: [96]  # MiB reserved for activations on the single GPU

  prompt_template:
  template_vars_default:
    enable_thinking: true
  reasoning: true          # required for tool call parsing; splits reasoning_content / content
  vision: true             # model is multimodal (Qwen3.5 vision tower)

  # --- tool / function calling parser ---
  tool_format: qwen3_coder
  tool_calls_in_reasoning: true

  use_dummy_models: true
  dummy_model_names: ["qwen3.8-27b"]

draft_model:
  draft_mode: disabled

sampling:
  override_preset: safe_defaults

logging:
  log_prompt: false
  log_generation_params: false
  log_requests: false
  log_chat_completion_requests: false

memory:
  cuda_malloc_async: true

developer:
  unsafe_launch: false

Note: the ghcr.io/theroyallab/tabbyapi image lacks python3.12-dev, which Triton needs for JIT compilation of cuda_utils.c. The entrypoint is overridden to install the dev headers and start the server in one shot:

bash

docker run --rm --gpus all -p 10404:5000 \
  -v './Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw:/app/models/Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw:ro' \
  -v './tabby-config-2.25.yml:/app/config.yml:ro' \
  --entrypoint bash ghcr.io/theroyallab/tabbyapi:latest \
  -c "apt-get update -qq && apt-get install -y --no-install-recommends python3.12-dev gcc && exec python3 main.py --host 0.0.0.0"

VRAM observed at 64k cache for this 2.25 build: ~15.6 GB (16 013 MiB) with cache_size=65536, cache_mode=Q8, max_seq_len=65536 (RTX 4090, single user) — vs 16.6 GB for the 2.75bpw twin at the same cache_size; the ~964 MiB delta matches the ~1.5 GB smaller weight footprint.

Build provenance

Quantizer: turboderp-org/exllamav3 (tag v1.4.2)
Host: Windows 11, Python 3.14.7, torch 2.11.0+cu128, CUDA 12.8, triton-windows 3.7.1
GPU: NVIDIA GeForce RTX 4090 (24 GB, sm 8.9), driver 610.88
Build wall time: 1 h 33 min 14 s end-to-end (per-module sum: 1 h 04 min) — pure EXL3 quant phase on RTX 4090.
Start / end timestamps: 2026-08-16 14:15:30 → 2026-08-16 15:48:44
Modules quantized: 70 (1 embed_tokens + 64 transformer + 1 norm + 1 lm_head + 3 mtp/projection)
Calibration source: exllamav3 v1.4.2 standard_cal_data (6 files, ~5.6 MB)
VRAM at inference (TabbyAPI, observed): ~15.6 GB (16 013 MiB) with cache_size: 65536, cache_mode: Q8, max_seq_len: 65536. Single user, RTX 4090 24 GB. 128k-cache figure not yet measured for this build (2.75bpw twin measured ~19.5 GB at 128k).

License

Apache License 2.0. See LICENSE — inherited from the upstream Blackfrost repo, which traces to Alibaba Cloud (Qwen authors).

WatchDG/Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw

Qwen3.8-27B-ABLITERATED-EXL3-2.25bpw