This release targets ExLlamaV3 and TabbyAPI. It was converted at the maximum nominal EXL3 bitrate supported by the converter, with the input embedding, output head, and vision tower protected at 16-bit precision.

Model details

Architecture: Qwen3.5 MoE multimodal conditional generation
Logical parameters: 35,107,181,936
Active parameters: approximately 3B per token
Hidden size: 2,048
Vocabulary size: 248,320
Transformer layers: 40
Full-attention layers: 10
Recurrent Gated DeltaNet layers: 30
Maximum context: 262,144 tokens
Vision parameters: 446,571,248
Native MTP layers: 0
Upstream revision: b46d88b3fc26062086e303e4f9d34f593c9470b4
License: Apache-2.0, inherited from the upstream model

The Hugging Face generic tensor viewer may miscount this checkpoint because EXL3 uses packed custom tensors. The logical parameter total above comes from the complete upstream tensor inventory; no language or vision layers were removed.

Quantization details

Property	Value
EXL3 nominal bitrate	8.00 bpw
Codebook	`mul1`
Output scales	Always enabled
Input embedding	BF16
Output head	Unquantized FP16 via `-hb 16`
Vision tensors	223 BF16 and 110 F32 tensors
Calibration rows	768
Calibration columns	4,096
Calibration tokens	3,145,728
Calibration seed	20260810
ExLlamaV3 format version	1.4.1
Indexed tensor bytes	36,757,084,872
Package size	34.32 GiB

Calibration emphasized code, debugging, tests, tool calls, structured data, technical material, reasoning and mathematics, multilingual technical text, general instruction data, and reference material. Construction used fixed revisions, document-hash deduplication, deterministic packing, and the actual BTL-4 tokenizer and chat template.

Measured benchmark

These are measured deployment results, not vendor estimates.

Hardware and runtime:

2x NVIDIA RTX 3090 24 GB
No active NVLink
ExLlamaV3 1.4.1 through TabbyAPI
Layer autosplit, tensor parallelism disabled
262,144-token FP16 KV cache
Four request slots
Vision enabled
CPU MoE offload disabled
Speculative decoding disabled

Workload	Result
Four concurrent generations, 512 output tokens each	18.925 s wall time
Aggregate four-slot decode throughput	108.216 tokens/s
Approximate per-request throughput at four slots	27.211 tokens/s
Minimum free VRAM during four-slot test	3,029 / 4,565 MiB
214,680-token prefill	1,470.31 tokens/s
248,013-token prefill	1,349.22 tokens/s
Decode after 248K prefill	63.46 tokens/s
Minimum free VRAM during 248K test	2,767 / 4,271 MiB
Vision test generation	104.76 tokens/s
Reasoning test generation	107.43 tokens/s

Functional acceptance:

Four concurrent request slots: pass
Exact structured JSON: pass
Parsed Qwen tool call: pass
Coding smoke test: 4/4 cases passed
Arithmetic reasoning smoke test: pass
248K middle-needle retrieval: exact match
Vision shape recognition and OCR: exact match

Throughput varies with prompt length, output length, sampler, slot occupancy, GPU topology, driver, and TabbyAPI/ExLlamaV3 versions. The functional checks above are focused deployment smoke tests, not a replacement for standardized academic evaluation suites.

Download

bash

hf download P4pps3n/BTL-4-EXL3-8bpw \
  --local-dir /path/to/models/BTL-4-EXL3-8bpw

TabbyAPI setup

Use a current TabbyAPI checkout with ExLlamaV3 1.4.1 or a compatible newer release. A tested example is included as tabby_config.yml.

Edit only the model path and network/auth settings for your environment:

yaml

model:
  model_dir: /path/to/models
  model_name: BTL-4-EXL3-8bpw
  backend: exllamav3

  max_seq_len: 262144
  cache_size: 262144
  cache_mode: FP16
  max_batch_size: 4

  tensor_parallel: false
  gpu_split_auto: true
  autosplit_reserve: [2048, 2048]
  cpu_moe_offload_layers: 0

  vision: true
  reasoning: true
  tool_format: qwen3_coder

Select GPUs before starting TabbyAPI. UUIDs are safer than unstable numeric indices:

bash

export CUDA_DEVICE_ORDER=FASTEST_FIRST
export CUDA_VISIBLE_DEVICES="GPU_UUID_1,GPU_UUID_2"
python /path/to/tabbyAPI/main.py --config /path/to/tabby_config.yml

Check the server:

bash

curl -fsS http://127.0.0.1:5004/health
curl -fsS http://127.0.0.1:5004/v1/model/list

The tested 256K FP16, four-slot profile uses roughly 41 GB across both cards. For systems with less aggregate VRAM, reduce max_batch_size before reducing model bitrate, context length, or cache precision. If necessary, test Q8 cache as a separate capacity profile.

OpenAI-compatible usage

Text example:

bash

curl http://127.0.0.1:5004/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "BTL-4-EXL3-8bpw",
    "messages": [{"role": "user", "content": "Explain delta networks."}],
    "template_vars": {"enable_thinking": false},
    "temperature": 0.2,
    "max_tokens": 512
  }'

Thinking can be enabled per request:

"template_vars": {"enable_thinking": true}

For tool calling, provide OpenAI-compatible tools and use TabbyAPI's qwen3_coder tool parser. The model's chat template emits XML tool blocks; TabbyAPI converts them into standard tool_calls responses.

Vision example in Python:

python

import base64
import requests

with open("image.png", "rb") as f:
    image = base64.b64encode(f.read()).decode("ascii")

payload = {
    "model": "BTL-4-EXL3-8bpw",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {
                "type": "image_url",
                "image_url": {"url": "data:image/png;base64," + image},
            },
        ],
    }],
    "template_vars": {"enable_thinking": False},
    "max_tokens": 512,
}

response = requests.post(
    "http://127.0.0.1:5004/v1/chat/completions",
    json=payload,
    timeout=300,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"])

Image embeddings consume tokens from the same shared context pool.

Notes and limitations

This is an EXL3 checkpoint and requires an ExLlamaV3-compatible runtime.
The full vision component is retained; preprocessor_config.json is required.
There is no native MTP component in BTL-4.
No speculative draft model is included.
Native tensor parallelism was not selected because the tested GPUs had no active NVLink; layer autosplit produced strong throughput and stable output.
-hb 16 is stored as unquantized FP16 by this ExLlamaV3 converter, not BF16.
The 256K FP16 profile was validated on two 24 GB GPUs. Smaller systems must tune cache precision, slot count, context, or model placement.

Acknowledgements

Base model: badtheorylabs/BTL-4
Quantization/runtime: ExLlamaV3
Serving: TabbyAPI