You can deploy this model and start earning money today!

XReyRobert/
Qwen3.8-27B-W4A16-AutoRound-INT8-Head-Embeddings

Name: Qwen3.8-27B-W4A16-AutoRound-INT8-Head-Embeddings
Availability: InStock

Views

No views yet

Qwen3.8-27B W4A16 AutoRound with INT8 Head and Embeddings

This checkpoint starts from dbirks/Qwen3.8-27B-W4A16-AutoRound and applies the serving recipe published by syv-ai/qwen38-27b-rtx3090.

The language-model linear layers remain the original AutoRound W4A16 compressed-tensors quantization. The two large untied vocabulary matrices are additionally quantized to symmetric INT8 with group size 128:

lm_head: INT8 group-128, relative round-trip error 0.0064
model.language_model.embed_tokens: INT8 group-128, relative round-trip error 0.0056

This saves approximately 2.6 GB of weight memory compared with the source W4A16 checkpoint, leaving more VRAM for KV cache and recurrent state pages on 24 GB GPUs.

Provenance

Base model: Qwen/Qwen3.8-27B
Base model revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Source quant: dbirks/Qwen3.8-27B-W4A16-AutoRound
Source checkpoint revision: 1f05c441c4e64ae0549de44fa9ea5a6d43610314
Transformation recipe: syv-ai/qwen38-27b-rtx3090
Recipe revision: e681a8f6b190cca1f86d58b8b333b64d014994d7
Output format: compressed-tensors pack-quantized
Body: symmetric INT4 W4A16, group size 128
Vocabulary matrices: symmetric INT8, group size 128

The exact transformation scripts and vLLM patch used are included in this repository.

Important vLLM requirement

The INT8 token embedding table requires a small vLLM patch. Stock vLLM 0.27.1 contains the dequantization kernel but does not pass the quantization config to Qwen3.5/Qwen3.8's VocabParallelEmbedding constructors.

Apply the included patch to the installed vLLM package:

bash

patch -p1 -d /path/to/site-packages/vllm \
  < vllm-qwen3_5-embed-quant.patch

The patch was written and verified against vLLM 0.27.1. Revalidate it after upgrading vLLM. Without this patch, the checkpoint is not expected to load correctly.

Text-only serving on a 24 GB GPU

The following profile mirrors the batch-oriented syv-ai recipe:

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve XReyRobert/Qwen3.8-27B-W4A16-AutoRound-INT8-Head-Embeddings \
  --served-model-name qwen3.8-27b-autoround-int8-head-embeddings \
  --language-model-only \
  --kv-cache-dtype fp8 \
  --max-model-len 150000 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.972 \
  --async-scheduling \
  --compilation-config '{"max_cudagraph_capture_size":64,"custom_ops":["+rms_norm","+silu_and_mul"]}' \
  --trust-remote-code

--language-model-only avoids loading the vision tower. The published syv-ai measurements use an RTX 3090 and describe 150k as the conservative deployment context; the reported single-request ceiling after requantization is around 195k. Actual capacity depends on the runtime version, CUDA allocator state, concurrency and prompt shape.

For single-user speculative decoding, use the MTP profile documented in the source recipe and ensure the second hunk of the included patch is applied.

Sampling and thinking control

Use the source model's recommended sampling parameters:

Mode	Temperature	Top-p	Top-k	Min-p	Presence penalty	Repetition penalty
Thinking	`1.0`	`0.95`	`20`	`0.0`	`0.0`	`1.0`
Instruct / non-thinking	`0.7`	`0.8`	`20`	`0.0`	`1.5`	`1.0`

Qwen3.8 supports reasoning_effort=low, medium, or xhigh; xhigh is the source-model default. For agentic sessions, enable thinking and preserve prior reasoning through the chat template:

{"enable_thinking": true, "preserve_thinking": true}

Keep the combined input, retained reasoning, and generated output inside the served context limit. The 150K profile above is smaller than the model's native 262,144-token position limit because it is sized for one 24 GB GPU.

Reproduction

Starting from the pinned source revision:

bash

python quant_lm_head.py /path/to/checkpoint
python quant_embed.py /path/to/checkpoint

The scripts rewrite the relevant safetensors shards, update model.safetensors.index.json, and add dedicated compressed-tensors config groups for lm_head and embed_tokens.

Validation performed

Safetensors/index consistency: 2,003 indexed tensors, 2,003 present
Missing indexed tensors: 0
Extra tensors in indexed shards: 0
lm_head relative round-trip error: 0.0064
embed_tokens relative round-trip error: 0.0056

Terminal-Bench 2.0 sanity check

The published artifact was exercised with Terminus 2 on three tasks (openssl-selfsigned-cert, headless-terminal, and count-dataset-tokens) using one RTX 3090, one concurrent trial, 32 CPU cores, 48 GiB RAM, and a 30-minute task timeout.

Effort	Verifier reward	Run wall	Visible output	vLLM output	Parser warnings	Decode	Prefill	Agent timeouts
`low`	3/3	36.5 min	43,988	60,696	18	46.9 tok/s	1,175 tok/s	0
`xhigh`	3/3	52.1 min	95,325	96,877	22	46.2 tok/s	1,168 tok/s	1

The xhigh timeout occurred after active generation and is retained as an observed benchmark outcome even though the verifier reward was already 1 for that task. Parser warnings indicate extra or discarded model text around tool actions; they do not automatically imply an incorrect tool call.

This is a serving and agent-path smoke check, not a broad quality ranking. Three tasks are too few for comparative accuracy claims. Refer to the source AutoRound model card for body-quantization evaluations and to the syv-ai repository for the original INT8 head/embedding serving benchmarks.

Credits

Qwen team for Qwen3.8-27B
dbirks for the W4A16 AutoRound checkpoint
syv-ai for the INT8 head/embedding transformation and vLLM serving recipe