uv run python quantization/quantize_qwen35_9b_gptq.py \
  --model-name Qwen/Qwen3.5-9B \
  --output-dir /workspace/lowbit-math-reasoning/experiments/models/Qwen3.5-9B-GPTQ-INT8 \
  --dataset-name zwhe99/DeepMath-103K \
  --dataset-config '' \
  --dataset-split train \
  --calibration-preset math_qa_cot \
  --question-column question \
  --answer-column r1_solution_1 \
  --text-column r1_solution_1 \
  --max-calibration-samples 128 \
  --max-seq-len 16384 \
  --bits 8 \
  --group-size 128 \
  --damp-percent 0.1

The current quantization script rewrites config.json after save_pretrained() so the exported checkpoint uses the same text-only qwen3_5_text layout as the working INT4 checkpoint.

Validation

This normalized-config checkpoint was re-evaluated on GSM8K and matched the original INT8 accuracy while improving throughput substantially.

Original INT8: EM 0.96, 105.98 tok/s
Fixed-config INT8: EM 0.96, 150.84 tok/s

Notes

This repository contains quantized weights only.
The checkpoint is intended for text-only evaluation.
vLLM loads this checkpoint as gptq_marlin.

lullyli/Qwen3.5-9B-GPTQ-INT8

Qwen3.5-9B-GPTQ-INT8

Quantization

Reproduction

Validation

Notes

lullyli/
Qwen3.5-9B-GPTQ-INT8