
Base model	Qwen/Qwen2.5-7B-Instruct
Parameters	7.61B
Architecture	GQA, 28 layers, 28 heads, 4 KV heads
Quantization	AWQ W4A16 asymmetric, group_size=128
Model size	5.3 GB
VRAM required	~6.5 GB (inference)

Quantization Details

Parameter	Value
Method	AWQ
Tool	llmcompressor
Bits	4
Group size	128
Scheme	W4A16_ASYM (4-bit weights, 16-bit activations, asymmetric)
Calibration dataset	Salesforce/wikitext (wikitext-103-raw-v1)
Calibration samples	128
Seed	42
Quantization time	1098s
Hardware	RunPod RTX 6000 Ada (48 GB)

Why Self-Quantized?

Pre-quantized checkpoints on HuggingFace typically have unknown calibration provenance â€” the dataset, sample count, seed, and group size are rarely documented. This checkpoint was self-quantized with controlled, documented settings to enable rigorous cross-method comparison (GGUF k-quant vs AWQ vs GPTQ) in a NeurIPS 2026 submission on quality-safety correlation under quantization.

Evaluation Results

Evaluation pending â€” quality and safety benchmarks will be run on this checkpoint and results updated here.

Other Quantization Formats

Format	Repository
Original FP16	Qwen/Qwen2.5-7B-Instruct
GPTQ 4-bit	Crusadersk/qwen2.5-7b-gptq-4bit

Prompt Template

<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Crusadersk/qwen2.5-7b-awq-4bit",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Crusadersk/qwen2.5-7b-awq-4bit")

messages = [{"role": "user", "content": "What is the capital of France?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Compatibility

Framework	Supported
Transformers	Yes
vLLM	Yes (AWQ backend)
llama.cpp	No (use GGUF format instead)
Ollama	No (use GGUF format instead)
Windows (native)	No â€” requires Linux/Docker

Reproduction

The full quantization pipeline â€” Dockerfiles, quantization scripts, and a 766-line engineering log documenting every platform failure and solution â€” is available at:

research/tr142/expansion/

in the Banterhearts repository.

Citation

bibtex

@misc{banterhearts2026qwen7bawq,
  title = {Self-Quantized Qwen 2.5 7B Instruct (AWQ 4-bit) for Quality-Safety Correlation Research},
  author = {Kadadekar, Sahil},
  year = {2026},
  url = {https://huggingface.co/Crusadersk/qwen2.5-7b-awq-4bit},
  note = {Part of the Banterhearts research program. NeurIPS 2026 submission.}
}

Acknowledgments

This work is part of the Chimera/Banterhearts technical-report program on deployment-time LLM behavior, quantization, refusal robustness, batching effects, and inference-stack reliability. Canonical public archive: Chimeraforge Reports; source context: github.com/Sahil170595/Banterhearts.