You can deploy this model and start earning money today!

amd/
Mixtral-8x7B-Instruct-v0.1-w4a16-llmcompressor

Name: Mixtral-8x7B-Instruct-v0.1-w4a16-llmcompressor
Availability: InStock

Views

No views yet

Mixtral-8x7B-Instruct-v0.1-w4a16-llmcompressor-v0.12.0

Model Overview

Model Architecture: MixtralForCausalLM
- Input: Text
- Output: Text
Source Model: Mixtral-8x7B-Instruct-v0.1
Supported Hardware: AMD EPYC (CPU inference)
Preferred Operating System: Linux
Inference Engine: vLLM v0.26.0
Quantization Framework: LLM Compressor v0.12.0
Quantization Method: 4-bit Weight-Only Quantization (W4A16)
Compatible Stack:
- ZenDNN v6.1.0
- ZenTorch v2.11.0.3
- PyTorch v2.11.0
- LLM Compressor v0.12.0
- vLLM v0.26.0

This is a quantized version of Mixtral-8x7B-Instruct-v0.1 created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Mixtral-8x7B-Instruct-v0.1 using LLM Compressor via the GPTQ algorithm. This reduces the model weights from 87.0 GiB to 22.8 GiB on disk (~74% reduction).

Method: 4-bit Weight-Only Quantization (W4A16)
Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128
Weights: INT4, symmetric, group-wise (group_size=128, actorder=static), stored as pack-quantized
Activations: BF16 (unquantized)
Group Size: 128
Kept in BF16: MoE router (block_sparse_moe.gate) and lm_head
Calibration: 128 examples from HuggingFaceH4/ultrachat_200k at a max sequence length of 2048

Both the attention projections and all eight per-expert gate_proj / up_proj / down_proj matrices are quantized. Only the tiny routing layer stays in BF16, which is why the on-disk reduction (~74%) is larger than for dense models: the experts dominate the parameter count.

Quantization command:

bash

numactl --physcpubind=0-95 python llm_compressor_quantize_and_run.py \
    --model_id mistralai/Mixtral-8x7B-Instruct-v0.1 \
    --save_dir ./Mixtral-8x7B-Instruct-v0.1-w4a16-llmcompressor-v0.12.0 \
    --recipe gptq \
    --scheme W4A16 \
    --run

Equivalent minimal steps:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization.gptq import GPTQModifier

model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
output_dir = "./Mixtral-8x7B-Instruct-v0.1-w4a16-llmcompressor-v0.12.0"

NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 2048

# Step 1: Load the BF16 model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Load calibration data. GPTQ is data-driven: it needs real activations
# to build the per-layer Hessians used to compensate the rounding error.
ds = load_dataset(
    "HuggingFaceH4/ultrachat_200k",
    split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]",
)
ds = ds.map(
    lambda example: {"text": "\n".join(m["content"] for m in example["messages"])},
    remove_columns=ds.column_names,
)

# Step 3: Define the W4A16 recipe. The preset implies group_size=128, which is
# valid here because Mixtral's 4096 and 14336 column counts are both divisible
# by 128. The router is skipped: it is tiny and wrong routing wrecks accuracy.
recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "lm_head",
        r"re:.*\.router$",
        r"re:.*\.router\..*",
        r"re:.*\.gate$",
        r"re:.*\.mlp\.gate$",
    ],
)

# Step 4: One-shot quantize with calibration and save in compressed-tensors format
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    processor=tokenizer,
)
model.save_pretrained(output_dir, save_compressed=True)
tokenizer.save_pretrained(output_dir)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

python

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Mixtral-8x7B-Instruct-v0.1-w4a16-llmcompressor-v0.12.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.11.0
zentorch==2.11.0.3
vllm==0.26.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

bash

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	W4A16 (this model)	Recovery
GSM8K (5-shot)	0.6406	0.6331	98.83%

Evaluation Command

bash

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Mixtral-8x7B-Instruct-v0.1-w4a16-llmcompressor-v0.12.0,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.11.0.3 / PyTorch v2.11.0. It may not load correctly on other versions.
CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
Accuracy Trade-off: 4-bit weight-only quantization is more aggressive than INT8. On GSM8K the model retains 98.83% of the BF16 baseline for a ~74% smaller memory footprint.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.