Tool: llm-compressor (vLLM project)
Format: compressed-tensors (auto-detected by vLLM)
Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)
Algorithm: AWQ (Activation-aware Weight Quantization)
Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr)
lm_head left in fp16 (preserves output quality)
Note: v_proj → o_proj smoothing is skipped on all 32 layers due to GQA shape mismatch (32 query heads / 8 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.

Usage with vLLM

bash

vllm serve KebalBaguette/Mistral-7B-Instruct-v0.3-abliterated-AWQ \
  --quantization compressed-tensors \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --max-model-len 8192

License

Apache 2.0, inherited from the upstream model and the abliterated base.