Weights are 3-bit AWQ values stored dequantized in BF16, so this loads with a stock AutoModelForCausalLM.from_pretrained — no AWQ kernels or custom classes. Every decoder-block linear holds at most 8 distinct values per group of 128 input channels; embeddings and lm_head are untouched (weight-only quantization).

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
m = AutoModelForCausalLM.from_pretrained(
    "ghostchoir/rebuttal_w3g128_awq", torch_dtype=torch.bfloat16, device_map="cuda")
tok = AutoTokenizer.from_pretrained("ghostchoir/rebuttal_w3g128_awq")

Quantization


Base model	`Qwen/Qwen3-1.7B` (BF16, 28 blocks, hidden 2048)
Scheme	weight-only, 3-bit, group size 128, asymmetric (zero point)
Scope	all `self_attn` / `mlp` linears in decoder blocks
Search	AWQ scale exponent (20-point grid) + clipping (stock per-output-channel search)

Search objective

The candidate is scored by the MSE at the output of the block being quantized:

inps ──► block i (candidate weights) ──► MSE vs block i unquantized

Note this is a block-wise objective. Stock AWQ scores at the output of the sub-module it perturbs (self_attn, mlp, or a single nn.Linear), which is a narrower scope than the whole block. Clipping is stock AWQ's per-output-channel threshold search.

Calibration


Dataset	`sumin99/Openr1-dataset-sampled-CMA`
Samples	first 4096 rows (of 32768)
Length	first 2048 tokens per row, one document per row (no concatenation)
Padding	right-padded with `<
Totals	8,131,131 real tokens, 3.07% padding, 3600/4096 rows truncated

Text is tokenized as-is — no chat template applied, no stripping (rows already carry ChatML markup).

Results

Perplexity at sequence length 2048. Bold is this model.

model	WikiText-2	Δ	OpenR1-CMA held-out	Δ
Qwen3-1.7B BF16 (unquantized)	16.7162	—	2.2045	—
RTN W3/g128 (no AWQ)	53.5499	+36.8337	4.4508	+2.2463
AWQ W3/g128 block-wise (d=0)	23.4184	+6.7022	2.6564	+0.4520
AWQ W3/g128 look-ahead (d=1)	25.0860	+8.3698	2.8304	+0.6260

Held-out slice = rows 4096–5119 of the calibration dataset (never seen by the search). RTN = round-to-nearest group quantization with no AWQ search.

Comparability caveat. The d=0 and d=1 arms differ in two ways, not one: the loss scope (block output vs next-block output) and the clipping search (stock per-output-channel vs a single global shrink scored look-ahead). The clipping difference does more work at 3-bit than at 4-bit, so the gap between these two arms should not be read as an ablation of the loss alone.