Views
No views yet
9d85770e5eb602322b4bceef55beda357e0bd0ca), an abliterated Qwen3.8-27B multimodal model.Third-party derivative, unevaluated by upstream. This is a community EXL3 build, not an official Qwen or Blackfrost release. Blackfrost's own model card states: "Any additional direction editing, fine-tuning, merging, pruning, quantization, or other weight change creates an artifact Blackfrost has not evaluated unless a new report explicitly states otherwise." That applies directly here — quality, refusal behavior, reasoning, tool use, vision/video behavior, and long-context stability of this specific 3.75 bpw quant have not been assessed by Blackfrost and may differ from the BF16 reference. This is a weight-level research checkpoint with a deliberately reduced refusal surface; it is not a safety-stock model and must not be represented, deployed, or evaluated as one.
Qwen3_5ForConditionalGeneration (hybrid linear + full attention, 64 layers, MTP)| Component | bpw | Notes |
|---|---|---|
| Transformer body (64 layers) | 3.75 | as requested via -b 3.75 |
embed_tokens | 16 | not quantized (exllamav3 stores embedding tables raw bf16, not via trellis) |
lm_head | 6 | -hb 6 — protects output quality |
| MTP (1 layer) | 4 | -mb 4 (exllamav3 default) |
| Vision tower (ViT) | 16 | -vb 16 — effectively unquantised |
| Norm tensors | 16 | unquantised |
mul1 (default)always| File | Size (B) | SHA-256 (16 hex) | Role |
|---|---|---|---|
chat_template.jinja | 10 446 | 6E0F10E53C40D7F3 | Jinja template (modified — see below) |
config.json | 4 623 | AAD7E008F29009F9 | architecture + quantization_config |
generation_config.json | 202 | E70C136C1B78DDC1 | eos/bos + sampling defaults |
LICENSE | 11 544 | BBEDC3FDA3305820 | Apache License 2.0 |
model.safetensors.index.json | 241 096 | 05625D5C63DC9D82 | tensor → shard map |
model-00001-of-00002.safetensors | 8 453 218 763 | 4B2B47D74C3C73C0 | weights shard 1 (7.87 GiB) |
model-00002-of-00002.safetensors | 7 646 718 010 | 3B1F4DEF788B2B0A | weights shard 2 (7.12 GiB) |
preprocessor_config.json | 391 | 5102CC0567B75A34 | vision/image preprocessor (Qwen2VL) |
quantization_config.json | 646 558 | D130BAE6C8E144DF | exllamav3 per-tensor bit/storage map (707 leaf modules) |
tokenizer.json | 12 809 320 | 0997F410C57A1F4E | HF fast tokenizer (BPE) |
tokenizer_config.json | 19 349 | 3E63E525D8309DCE | tokenizer metadata + embedded chat template |
video_preprocessor_config.json | 386 | 00BD47A5EAAF8760 | video preprocessor (Qwen3VL) |
config.json — appended a quantization_config block (ExL3 specifics). All upstream fields preserved unchanged.quantization_config.json — exllamav3-generated per-tensor storage table; not present in the BF16 source (which has no quantization).chat_template.jinja — added or message.role == "developer" in three places so that the template also accepts role=developer as an alias for role=system. The upstream template only recognises system/user/assistant/tool; the developer role is used by some OpenAI-style APIs for the higher-priority instruction.generation_config.json, tokenizer.json, tokenizer_config.json, preprocessor_config.json, video_preprocessor_config.json) are byte-identical to the Blackfrost source.tabby-config-3.75.yml used to serve this model on a 24 GB GPU (RTX 4090 verified):1network:
2 host: 0.0.0.0
3 port: 5000
4 disable_auth: true # only safe behind localhost / a trusted network
5
6model:
7 model_dir: /app/models
8 model_name: Qwen3.8-27B-ABLITERATED-EXL3-3.75bpw
9
10 # --- KV cache (24 GB budget) ---
11 cache_size: 131072 # max tokens of KV cache
12 cache_mode: Q8 # 8-bit cache; halves VRAM vs FP16, < 10% speed cost
13 max_seq_len: 131072 # cap request length to the cache
14
15 # --- EXL3-specific (optional) ---
16 chunk_size: 2048 # prompt ingestion chunk size
17 output_chunking: true # allocate KV cache in chunks as needed (saves VRAM)
18 inline_model_loading: true # allow model swap by name in request body
19
20 # --- multi-GPU / single-GPU split ---
21 gpu_split_auto: true
22 autosplit_reserve: [96] # MiB reserved for activations on the single GPU
23
24 # --- chat template / parsing ---
25 prompt_template: # use the model's embedded template
26 template_vars_default:
27 enable_thinking: true # reasoning on by default
28 reasoning: true # required for tool call parsing; splits reasoning_content / content
29 vision: true # model is multimodal (Qwen3.5 vision tower)
30
31 # --- tool / function calling parser ---
32 tool_format: qwen3_coder
33 tool_calls_in_reasoning: true
34
35 # --- model-name aliasing ---
36 use_dummy_models: true
37 dummy_model_names: ["qwen3.8-27b"] # short alias exposed via /v1/models
38
39draft_model:
40 draft_mode: disabled # no speculative decoding
41
42sampling:
43 override_preset: safe_defaults # provides sampling fallbacks for OAI clients that don't send params
44
45logging:
46 log_prompt: false
47 log_generation_params: false
48 log_requests: false
49 log_chat_completion_requests: false
50
51memory:
52 cuda_malloc_async: true # torch allocator; recommended for EXL3
53
54developer:
55 unsafe_launch: falseghcr.io/theroyallab/tabbyapi image lacks python3.12-dev, which Triton needs for JIT compilation of cuda_utils.c. The entrypoint is overridden to install the dev headers and start the server in one shot:1docker run --rm --gpus all -p 10404:5000 \
2 -v './Qwen3.8-27B-ABLITERATED-EXL3-3.75bpw:/app/models/Qwen3.8-27B-ABLITERATED-EXL3-3.75bpw:ro' \
3 -v './tabby-config-3.75.yml:/app/config.yml:ro' \
4 --entrypoint bash ghcr.io/theroyallab/tabbyapi:latest \
5 -c "apt-get update -qq && apt-get install -y --no-install-recommends python3.12-dev gcc && exec python3 main.py --host 0.0.0.0"cache_size=131072, cache_mode=Q8, max_seq_len=131072.v1.4.2)LICENSE — inherited from the upstream Blackfrost repo, which traces to Alibaba Cloud (Qwen authors).