The base abliterated model was only published in BF16 and GGUF. This AWQ quant is ~7 GB VRAM and serves natively in vLLM with the Marlin kernel for fast inference.

Quantization details

Tool: llm-compressor (vLLM project)
Format: compressed-tensors (auto-detected by vLLM)
Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)
Algorithm: AWQ (Activation-aware Weight Quantization)
Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr)
lm_head left in fp16 (preserves output quality)
Note: v_proj → o_proj smoothing is skipped on all 40 layers due to GQA shape mismatch (32 query heads / 8 KV heads). This is standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.

Usage with vLLM

bash

vllm serve KebalBaguette/Mistral-Nemo-Instruct-2407-abliterated-AWQ \
  --quantization compressed-tensors \
  --max-model-len 8192

License

Apache 2.0, inherited from Mistral Nemo and the abliterated base.