Views
No views yet
Qwen/Qwen3-1.7B quantized to W4G128 with CBQ (Cross-Block Quantization,
arXiv:2312.07950), then dequantized back to
bf16.This is a dense bf16 checkpoint, not a packed low-bit one. Every weight has been round-tripped through the W4G128 grid, so it carries the accuracy of W4G128 quantization, but the file itself is an ordinaryQwen3ForCausalLMin bfloat16 with the same tensor names and shapes as the base model. It loads in vLLM or transformers with no custom kernel, noquantization_config, and no code change — and it gives no speed or memory benefit over the base model. Use it to evaluate what W4G128 costs in quality.
1from vllm import LLM
2llm = LLM("ghostchoir/rebuttal_w4g128_cbq") # dtype is bfloat16 per config.json| bit-width | 4 |
| granularity | group-128 (--w_group_size 128) |
| scheme | asymmetric, weight-only (activations stay bf16) |
| CBQ settings | group-tuned (see below) |
| CBD window / overlap | 2 / 1 |
| epochs per window | 3 |
| LoRA-Rounding rank | 5 |
| CFP weight truncation | off |
Delta_W hard-round fraction | 1.0 |
| held-out perplexity | 2.1147 (bf16 baseline 2.0084, delta +0.1063) |
| mean per-token KL(FP || quant) | 0.05527 over 63406 tokens |
Delta_W initialisation, and
the direction of the Algorithm 1 objective), and every such choice is documented with
the measurement behind it. Two settings differ from the paper deliberately:
group-wise rather than per-channel quantization where noted, and the calibration set
above.--cfp_weight off and
--hard_round_frac 1.0 rather than the paper's values. Both paper settings were
measured worse at group-128: CFP weight truncation costs ~+2.6 perplexity because
at group-128 an outlier only inflates its own 128-wide group, and rounding
Delta_W only in the late phase leaves most of the optimization on a near-lossless
(degenerate) objective.