Self-quantized GPTQ 4-bit checkpoint of Qwen/Qwen2.5-7B-Instruct with fully documented calibration provenance.
Created as part of the Banterhearts research program investigating quality-safety correlation under quantization for consumer LLM deployment.
Base model
Qwen/Qwen2.5-7B-Instruct
Parameters
7.61B
Architecture
GQA, 28 layers, 28 heads, 4 KV heads
Quantization
GPTQ 4-bit, group_size=128
Model size
5.3 GB
VRAM required
~6.5 GB (inference)
Quantization Details
Parameter
Value
Method
GPTQ
Tool
gptqmodel
Bits
4
Group size
128
Scheme
Symmetric (4-bit, INT32 packing)
Calibration dataset
allenai/c4 (en, shard 1 of 1024)
Calibration samples
128
Seed
42
Quantization time
495s
Hardware
RunPod RTX 6000 Ada (48 GB)
Why Self-Quantized?
Pre-quantized checkpoints on HuggingFace typically have unknown calibration provenance — the dataset, sample count, seed, and group size are rarely documented. This checkpoint was self-quantized with controlled, documented settings to enable rigorous cross-method comparison (GGUF k-quant vs AWQ vs GPTQ) in a NeurIPS 2026 submission on quality-safety correlation under quantization.
Evaluation Results
Evaluation pending — quality and safety benchmarks will be run on this checkpoint and results updated here.
1from transformers import AutoModelForCausalLM, AutoTokenizer
23model = AutoModelForCausalLM.from_pretrained(4"Crusadersk/qwen2.5-7b-gptq-4bit",5 device_map="auto",6)7tokenizer = AutoTokenizer.from_pretrained("Crusadersk/qwen2.5-7b-gptq-4bit")89messages =[{"role":"user","content":"What is the capital of France?"}]10inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)11outputs = model.generate(inputs, max_new_tokens=64, do_sample=False)12print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Inference requirements: pip install gptqmodel (Linux only) or optimum+auto-gptq
Windows users: GPTQ inference requires gptqmodel which only builds on Linux. Use Docker or WSL2.
Compatibility
Framework
Supported
Transformers
Yes
vLLM
Yes (GPTQ backend)
llama.cpp
No (use GGUF format instead)
Ollama
No (use GGUF format instead)
Windows (native)
No — requires Linux/Docker
Reproduction
The full quantization pipeline — Dockerfiles, quantization scripts, and a 766-line engineering log documenting every platform failure and solution — is available at:
1@misc{banterhearts2026qwen7bgptq,
2 title = {Self-Quantized Qwen 2.5 7B Instruct (GPTQ 4-bit) for Quality-Safety Correlation Research},
3 author = {Kadadekar, Sahil},
4 year = {2026},
5 url = {https://huggingface.co/Crusadersk/qwen2.5-7b-gptq-4bit},
6 note = {Part of the Banterhearts research program. NeurIPS 2026 submission.}
7}
Acknowledgments
This work is part of the Chimera/Banterhearts technical-report program on deployment-time LLM behavior, quantization, refusal robustness, batching effects, and inference-stack reliability. Canonical public archive: Chimeraforge Reports; source context: github.com/Sahil170595/Banterhearts.
