The accuracy gap between RTN and GPTQ for INT8 is typically only 0.1–0.5 % perplexity — calibration mostly matters at INT4 and below. Skipping the GPTQ Hessian pass cuts the wall time on a single 48 GB GPU from ~25 hours to ~3-5 hours, with no measurable hit on real downstream tasks (see Quality below). For a 234 B-parameter MoE this is a worthwhile trade.

Recipe

python

from llmcompressor.modifiers.quantization import QuantizationModifier
from compressed_tensors.quantization import QuantizationArgs, QuantizationScheme

w8a16 = QuantizationScheme(
    targets=["Linear"],
    weights=QuantizationArgs(
        num_bits=8,
        type="int",
        symmetric=True,
        group_size=128,
        strategy="group",
        dynamic=False,
        observer="memoryless_minmax",
    ),
)

recipe = QuantizationModifier(
    config_groups={"w8a16": w8a16},
    ignore=[
        "lm_head",            # output head — no quant
        "re:.*router.*",      # MoE expert routers — must stay precise
        "re:.*\\.gate\\b",    # router gate layers
        "re:.*embed_tokens.*",
    ],
)

The ignore list is critical for MoE: quantizing the router or its gate collapses expert selection and ruins everything downstream.

Source: operationrange/MiniMax-M2.7-BF16 (our exact BF16 dequant of the upstream FP8 checkpoint).

Files

~26 shards model-NNNNN-of-NNNNN.safetensors (≈ 130 GB total)
model.safetensors.index.json
config.json with compression_config describing the W8A16 scheme
recipe.yaml — the llmcompressor recipe used
tokenizer + custom modeling .py files

Inference

vLLM

bash

python -m vllm.entrypoints.openai.api_server \
    --model operationrange/MiniMax-M2.7-8bit \
    --quantization compressed-tensors \
    --tensor-parallel-size 8 \
    --trust-remote-code

SGLang

bash

python -m sglang.launch_server \
    --model-path operationrange/MiniMax-M2.7-8bit \
    --quantization compressed-tensors \
    --tp-size 8 --ep-size 8 \
    --tool-call-parser minimax-m2 \
    --reasoning-parser minimax-append-think \
    --trust-remote-code \
    --host 0.0.0.0 --port 8080 \
    --mem-fraction-static 0.85

Fits on 8× 24 GB Ampere (e.g. RTX A5000) with TP=8, EP=8, leaving room for KV cache. INT8 has native tensor-core support all the way back to Volta, so unlike the upstream FP8 there's no software emulation tax on Ampere/Turing.

Quality

Recommended sampling (inherited from upstream MiniMax-M2.7 model card):

temperature = 1.0
top_p       = 0.95
top_k       = 40

Tool-calling and <think> reasoning are preserved — the router and embeddings are kept at full precision; only the heavy Linear layers (attention QKV/O, MLP up/gate/down, MoE expert weights) are INT8.

Provenance

234 B total / ~45.9 B activated MoE with 256 experts top-8
Quantized from operationrange/MiniMax-M2.7-BF16 using llmcompressor 0.10 and compressed-tensors 0.10.
Script: scripts/quant/quantize_rtn_w8a16.py

License

Inherits the MiniMax-M2 license from the upstream model. Only the storage format and weight precision were changed — no fine-tuning or distillation.