This is a GPTQ W4A16 (4-bit weight-only) quantization of Qwen/Qwen3.6-27B, a 27B parameter hybrid linear-attention / full-attention multimodal model with vision, native tool calling, and chain-of-thought reasoning capabilities.

Quantization was performed using llm-compressor (v0.10.1.dev141) with the GPTQModifier, producing a compressed-tensors pack-quantized format that maps directly to vLLM's MarlinLinearKernel (fused INT4 GEMM) for maximum throughput. This quantization is specifically optimized for SDLC agent workloads (code generation, multi-turn tool calling) through task-specific calibration.

Model Details

Property	Value
Base Model	Qwen/Qwen3.6-27B
Quantization Method	GPTQ W4A16 (asymmetric, group_size=32)
Weight Precision	INT4 (4-bit, zero-point quantization)
Activation Precision	FP16 (weight-only quantization)
Group Size	32 (fine-grained, high quality)
Quantization Library	llm-compressor 0.10.1.dev141
Format	compressed-tensors (pack-quantized)
Architecture	Qwen3_5ForConditionalGeneration
LM Layers	64 (48 linear_attention + 16 full_attention, hybrid)
Hidden Size	5120
Intermediate Size	17408
Attention Heads	24 (query), 4 (key/value, GQA)
Head Dimension	256
Vocabulary Size	248,320
Context Window	262,144 tokens (262K)
Vision Encoder	Qwen3_VisionTransformer (27 layers, hidden_size=1152, patch_size=16)
Vision Projector	spatial_merge_size=2
Quantized Components	Text decoder Linear layers only
Preserved in BF16	Full vision encoder (all 27 layers + merger), linear_attn projection layers (in_proj_a/b), lm_head, embed_tokens
Model Size	~20.1 GB
Calibration	2,048 task-specific samples (code + tool-calling + general)

Quantization Details

Recipe

yaml

default_stage:
  default_modifiers:
    GPTQModifier:
      config_groups:
        group_0:
          targets: [Linear]
          weights:
            num_bits: 4
            type: int
            symmetric: false          # Asymmetric: uses zero-points (int8)
            group_size: 32            # Fine-grained: 32-element scaling groups
            strategy: group
            actorder: static          # Static activation ordering
            observer: memoryless_minmax
      ignore:
        - lm_head
        - re:.*visual.*              # Full vision encoder preserved in BF16
        - re:.*linear_attn\.in_proj_a  # Linear attn projections (output_size=48)
        - re:.*linear_attn\.in_proj_b
      block_size: 128
      dampening_frac: 0.005          # Tight Hessian (5× default)
      actorder: static
      offload_hessians: false

Calibration Dataset

2,048 task-specific samples formatted with the Qwen3.6 ChatML template (<|im_start|>system\n...<|im_end|>). Task-specific calibration aligns the GPTQ Hessian estimation with the model's deployment distribution — particularly important for instruct models.

Source	Domain	Samples	HF ID
Magicoder-Evol-Instruct	Code generation	800	`ise-uiuc/Magicoder-Evol-Instruct-110K`
APIGen-MT-5k	Multi-turn tool calling	500	`Salesforce/APIGen-MT-5k`
When2Call	Tool abstention	300	`nvidia/When2Call`
Pile-10k	General text (filler)	~500	`NeelNanda/pile-10k`
Total		~2,100

Quality Benchmarks

All benchmarks on wikitext-2-raw-v1 (test split), 639 non-overlapping chunks of 512 tokens, 296,434 tokens scored. Methodology consistent with llama.cpp ./perplexity -c 512 and the academic quantization literature. BF16 baseline PPL = 8.7116.

Comparative Quality

Model	PPL	Δ vs BF16	Mean KLD	Median KLD	Same-top-p	Δp RMS	Size
BF16 base	8.7116	—	—	—	—	—	~54 GB
This model (V4 g32 asym)	8.8761	+1.89%	0.0945	0.0201	92.75%	0.037	~20.1 GB

PPL = Perplexity (lower is better). Mean KLD = mean KL-divergence from BF16 output distribution (lower is better). Same-top-p = fraction of tokens where quantized model agrees with BF16 on the top predicted token (higher is better).

Throughput Benchmarks

Benchmarks measured on NVIDIA RTX 5090 (32 GB VRAM, sm120 / Blackwell) running vLLM with quantization: compressed-tensors, kv_cache_dtype: fp8_e4m3, max_model_len: 65536, single GPU (TP=1).

Single Request (concurrency=1)

20 requests, max_tokens=1024, temperature=0.9. Aggregate throughput is wall-clock total tokens / total wall time.

Metric	Value
Aggregate throughput	94.5 tok/s
Per-request throughput	min 39.6 / avg 91.2 / max 97.8 tok/s
Average latency (TTFT + gen)	7.97 s
Min / Max latency	1.49 s / 10.72 s
Completion tokens	min 141 / avg 753 / max 1024
Success rate	20/20 (100%)
Wall time	159.5 s

16 Concurrent Requests

160 total requests, max_tokens=1024, temperature=0.9.

Metric	Value
Aggregate throughput	835.6 tok/s
Per-request throughput	min 17.4 / avg 54.4 / max 68.3 tok/s
Average latency	14.69 s
Min / Max latency	1.94 s / 23.05 s
Completion tokens	min 93 / avg 807 / max 1024
Success rate	160/160 (100%)
Wall time	154.5 s

Hardware Requirements

GPU	VRAM	Recommended `gpu_memory_utilization`	Max Context
RTX 5090	32 GB	0.95	131,972 tokens
RTX PRO 6000 / A100 / H100	80–96 GB	0.92	262,133 tokens (full)
H200	141 GB	0.85	262,144+ tokens (with rope-scaling)

Minimum: 1× GPU with ≥32 GB VRAM (with reduced context window and fp8 KV cache).

The compressed-tensors pack-quantized format uses MarlinLinearKernel (fused INT4 GEMM) — requires NVIDIA GPU sm75+ (Turing and later). Works on: Ampere (A100, RTX 3090), Ada (RTX 4090), Hopper (H100, H200), Blackwell (RTX 5090, RTX PRO 6000).

Usage with vLLM

Tested with: vllm/vllm-openai:cu130-nightly-fe9c3d6c5f66c873d196800384ed6880687b9e52 (vLLM v0.19.2rc1.dev134)

Docker Deployment (example with rope 384k context)

bash

docker run -d --name vllm-qwen36-27b \
  --runtime=nvidia --gpus '"device=0"' \
  -p 8000:8000 \
  -v /path/to/model:/workspace/model \
  --ipc=host --shm-size=16g \
  --restart unless-stopped \
  vllm/vllm-openai:cu130-nightly-fe9c3d6c5f66c873d196800384ed6880687b9e52 \
  /workspace/model \
  --host 0.0.0.0 --port 8000 \
  --quantization compressed-tensors \
  --dtype float16 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 393216 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 8 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8_e4m3 \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --limit-mm-per-prompt '{"image":4}' \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --hf-overrides '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":1.5,"original_max_position_embeddings":262144}}}'

Example vLLM Configuration (YAML)

This configuration is deployed and verified on an RTX 5090 (32 GB):

yaml

# -- Model & Server ----------------------------------------------------------
model: /workspace/model
host: "0.0.0.0"
port: 8000
served_model_name: "qwen36-27b"
trust_remote_code: true
tensor_parallel_size: 1

# -- Quantization ------------------------------------------------------------
quantization: compressed-tensors

# -- Data Type ---------------------------------------------------------------
dtype: float16

# -- Context & Batching ------------------------------------------------------
max_model_len: 393216
max_num_batched_tokens: 8192
max_num_seqs: 8
enable_chunked_prefill: true

# -- Memory ------------------------------------------------------------------
gpu_memory_utilization: 0.95
enable_prefix_caching: true
kv_cache_dtype: fp8_e4m3

# -- Multi-Modal -------------------------------------------------------------
limit_mm_per_prompt:
  image: 4

# -- Tool Calling & Reasoning ------------------------------------------------
# qwen3_xml parser: handles <tool_call>...</tool_call> native format.
# Do NOT use qwen3_coder — it breaks on code containing angle brackets.
enable_auto_tool_choice: true
tool_call_parser: qwen3_xml
reasoning_parser: qwen3

# -- Thinking Mode -----------------------------------------------------------
# Disabled server-wide. Clients can override per-request:
#   extra_body={"chat_template_kwargs": {"enable_thinking": true}}
default_chat_template_kwargs:
  enable_thinking: false

# -- RoPE (REQUIRED for Qwen3.6 long context) --------------------------------
hf_overrides:
  text_config:
    rope_parameters:
      mrope_interleaved: true
      mrope_section: [11, 11, 10]
      rope_type: yarn
      rope_theta: 10000000
      partial_rotary_factor: 0.25
      factor: 1.5
      original_max_position_embeddings: 262144

# -- Misc --------------------------------------------------------------------
disable_custom_all_reduce: true
mm_encoder_tp_mode: data
mm_processor_cache_type: shm

Critical: The --hf-overrides / hf_overrides block with RoPE parameters is required for correct long-context behaviour. Without it, the model's YaRN scaling is not applied and context beyond ~256K tokens degrades.

Inference Test

bash

# Text completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen36-27b","messages":[{"role":"user","content":"What is 2+2? One word."}],"max_tokens":10}'

# Tool calling test
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen36-27b",
    "messages": [{"role":"user","content":"What is the weather in Prague?"}],
    "tools": [{"type":"function","function":{"name":"get_weather","description":"Get weather","parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}}}],
    "tool_choice": "auto",
    "max_tokens": 256
  }'

Transformers / Python

python

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model_id = "gratex/Qwen3.6-27B-W4A16-g32-asym"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
)

messages = [{"role": "user", "content": "Explain GPTQ quantization in two sentences."}]
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Files in This Repository

File	Size	Description
`model.safetensors`	~20.1 GB	Quantized model weights (INT4 LM + BF16 vision)
`config.json`	—	Model configuration with `quantization_config`
`tokenizer.json`	—	Qwen3 vocabulary (248,320 tokens)
`tokenizer_config.json`	—	Tokenizer config with ChatML chat template
`chat_template.jinja`	—	Qwen3.6 native chat template (ChatML format)
`generation_config.json`	—	Default generation parameters
`preprocessor_config.json`	—	Image preprocessor configuration
`merges.txt`	—	BPE merge rules
`vocab.json`	—	Vocabulary file
`recipe.yaml`	—	llm-compressor quantization recipe
`LICENSE`	—	Apache 2.0 License

License

This quantization is released under the Apache 2.0 License, following the base model's license.

The base model Qwen/Qwen3.6-27B is licensed under Apache 2.0.

See LICENSE for the full license text.

Citation

If you use this model in your research or production systems, please cite:

bibtex

@misc{qwen36-27b-gptq-w4a16-g32-asym,
  title  = {Qwen3.6-27B GPTQ W4A16 g32 Asymmetric Quantization},
  author = {Gratex International},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/gratex/Qwen3.6-27B-W4A16-g32-asym}},
  note   = {Quantized with llm-compressor 0.10.1.dev141, task-specific calibration}
}

Acknowledgments

This quantization was produced using hardware and infrastructure provided by Gratex International, a.s.

Original Model: Qwen/Qwen3.6-27B
Quantization Tool: llm-compressor
Quantization Format: compressed-tensors
Deployment Engine: vLLM