Hadamard-Lloyd Weight Quantization (Q5) of Jackrong/Qwopus3.6-27B-v2 — a Qwen3.5-based reasoning-distilled model with hybrid attention, MTP head, and vision tower.


Source weights	58 GB BF16 (15 shards)
HLWQ Q5+INT4 weights	18.3 GB (3.2× smaller)
Inference speed	23 tok/s sustained
KV cache at 200 tok	9.5 MB (HLWQ Q3) vs ~13 MB FP16
Quality	100% token-match vs FP16 baseline
MTP draft accuracy	79.0% top-1 / 97.5% top-5

Benchmarks

Measured on a single RTX 6000 Pro Blackwell (96 GB) with transformers main, BF16 path, SDPA attention.

Inference speed

Weight compression

KV cache compression

MTP head accuracy (preservation under quantization)

The MTP head is loaded raw BF16 from the source checkpoint and queried against the HLWQ-quantized main model. 79% top-1 acceptance means the MTP draft matches the quantized model's greedy output for ~4 of every 5 tokens — strong evidence that Q5 + INT4 preserved the semantic surface that MTP was trained against.

Architecture

Qwen3_5ForConditionalGeneration — hybrid linear+full attention, multimodal, MTP head
64 text layers: 48 linear_attention + 16 full_attention (pattern [L,L,L,F]×16)
head_dim=256, hidden_size=5120, GQA (24 Q heads / 4 KV heads)
1 MTP layer (mtp_num_hidden_layers=1) for speculative decoding
Vision tower kept in BF16 (multi-modal projector untouched)
496 Linears PQ5-quantized → 400 of those converted to torchao INT4 (non-128-aligned shapes stay PQ5-dequant BF16)

Pipeline

text

BF16 weights (58 GB)
        │
        ▼
HLWQ Q5: Hadamard rotation + Lloyd-Max 32-centroid quantization (block=128)
        │
        ▼
Dequantize per-Linear → BF16
        │
        ▼
torchao Int4WeightOnlyConfig(group_size=128) on aligned text Linears
        │
        ▼
HLWQ Q3 KV cache during generation (bit-packed, head_dim=256)
        │
        ▼
Generation at ~23 tok/s on consumer-class GPU

Quick Start

pip install polarquant

python

import polarengine_vllm  # auto-registers with transformers
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "caiovicentino1/Qwopus3.6-27B-v2-HLWQ-Q5"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, device_map="auto", dtype="bfloat16")

inputs = tok.apply_chat_template(
    [{"role": "user", "content": "Explain quantum entanglement in two paragraphs."}],
    add_generation_prompt=True, return_tensors="pt", return_dict=True,
).to(model.device)

out = model.generate(**inputs, max_new_tokens=300, do_sample=False)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

GPU sizing

GPU	VRAM	Fits	Notes
RTX 6000 Pro Blackwell	96 GB	✅	Validated at 23 tok/s
A100 80 GB	80 GB	✅	Full speed expected
A100 40 GB	40 GB	✅	Full speed expected
RTX 6000 Ada	48 GB	✅	Full speed expected
RTX 4090	24 GB	⚠️	Tight, use HLWQ Q3 KV
RTX 3090	24 GB	⚠️	Tight, use HLWQ Q3 KV

Methodology notes

Weight quantization: each nn.Linear is rotated by a 128×128 Walsh-Hadamard matrix, scaled by √128, then assigned to the nearest of 32 Lloyd-Max centroids (computed for a unit Gaussian). Codes are bit-packed (5 bits → 0.625 byte/weight).
KV cache quantization: per-head Hadamard 256×256 + 8-centroid (Q3) Lloyd-Max with bit-packing (3 bits / scalar). A 128-token residual buffer holds recent tokens unquantized for accuracy.
Hybrid handling: the 48 linear-attention layers receive a hybrid cache layer passthrough (no KV quantization needed — recurrent state instead).
Multimodal prefix: weights re-prefixed with model.language_model. to match the base model's safetensors key layout.

References

Paper: arXiv:2603.29078
Code: github.com/caiovicentino/eoq-quantization
Base model: Jackrong/Qwopus3.6-27B-v2
Original GGUF release: Jackrong/Qwopus3.6-27B-v2-MTP-GGUF