Tool: llm-compressor (vLLM project)
Format: compressed-tensors (auto-detected by vLLM)
Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)
Algorithm: AWQ (Activation-aware Weight Quantization)
Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr)
lm_head left in fp16 (preserves output quality)
Note: v_proj → o_proj smoothing is skipped on all 32 layers due to GQA shape mismatch (32 query heads / 8 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.

Usage with vLLM

bash

vllm serve KebalBaguette/aya-expanse-8b-abliterated-AWQ \
  --quantization compressed-tensors \
  --tool-call-parser hermes \
  --enable-auto-tool-choice \
  --max-model-len 8192

License

CC-BY-NC-4.0 — non-commercial use only. Inherited from Cohere Aya Expanse 8B and lenML's abliterated derivative. This abliterated derivative is for research, personal, and non-commercial evaluation use only. For commercial deployment, contact Cohere for licensing, or use an Apache-licensed alternative (e.g. Mistral 7B / Nemo).