You can deploy this model and start earning money today!

amd/
gpt-oss-20b-BF16-da8w8-torchao-v0.17.0

Name: gpt-oss-20b-BF16-da8w8-torchao-v0.17.0
Availability: InStock

Views

No views yet

gpt-oss-20b-da8w8-torchao-v0.17.0

Model Overview

Model Architecture: GptOssForCausalLM (Mixture-of-Experts)
- Input: Text
- Output: Text
Source Model: gpt-oss-20b-BF16
Supported Hardware: AMD EPYC (CPU inference)
Preferred Operating System: Linux
Inference Engine: vLLM v0.22.0
Quantization Framework: TorchAO v0.17.0
Quantization Method: 8-bit Dynamic Activation, 8-bit Weight Quantization, Symmetric
- Dense nn.Linear layers: per-tensor activations / per-row weights
- MoE expert weights (experts.gate_up_proj, experts.down_proj): per-row activations / per-row weights via FqnToConfig
- Skipped (kept in BF16): lm_head, router
Compatible Stack:
- ZenDNN v6.0.0
- zentorch v2.11.0.1
- PyTorch v2.11.0
- TorchAO v0.17.0
- vLLM v0.22.0

[!NOTE] zentorch v2.11.0.1 for PyTorch v2.11.0 has to be built from source.

Quantization

The model was produced using torchao. Both activations and weights are quantized to INT8 with symmetric mapping; activation scales are computed dynamically at runtime per token. The MoE expert tensors are quantized in a second pass with per-row granularity on both activations and weights.

python

import os
from collections import OrderedDict

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig
from torchao.quantization import (
    Int8DynamicActivationInt8WeightConfig,
    quantize_,
)
from torchao.quantization.granularity import PerRow
from torchao.quantization.quant_api import FqnToConfig
from torchao.quantization.quant_primitives import MappingType

MODEL_ID   = "unsloth/gpt-oss-20b-BF16"
OUTPUT_DIR = "amd/gpt-oss-20b-da8w8-torchao-v0.17.0"
os.makedirs(OUTPUT_DIR, exist_ok=True)

# Pass 1: standard dynamic-act / weight INT8 for nn.Linear layers.
# Skip lm_head and router (kept in BF16).
ao_config = Int8DynamicActivationInt8WeightConfig(
    version=2,
    act_mapping_type=MappingType.SYMMETRIC,
)
quantization_config = TorchAoConfig(
    ao_config,
    modules_to_not_convert=["lm_head", "router"],
)

quantized_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype=torch.bfloat16,
    device_map="cpu",
    quantization_config=quantization_config,
    trust_remote_code=True,
)

# Pass 2: quantize MoE expert weights that the TorchAoConfig pass skipped
# because they live as nn.Parameter tensors, not nn.Linear modules.
# Use per-row granularity on activations and weights for expert tensors.
ao_config_experts = Int8DynamicActivationInt8WeightConfig(
    version=2,
    act_mapping_type=MappingType.SYMMETRIC,
    granularity=(PerRow(dim=-1), PerRow(dim=1)),
)

# Match the MoE expert parameter tensors via their fully-qualified names.
expert_fqn_config = FqnToConfig(
    fqn_to_config=OrderedDict({
        r"re:.*\.experts\.gate_up_proj$": ao_config_experts,
        r"re:.*\.experts\.down_proj$":    ao_config_experts,
    })
)

quantize_(quantized_model, expert_fqn_config, filter_fn=None)

quantized_model.save_pretrained(OUTPUT_DIR)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.save_pretrained(OUTPUT_DIR)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
out = quantized_model.generate(**inputs, max_new_tokens=30, cache_implementation="static")
print(tokenizer.decode(out[0], skip_special_tokens=True))

[!NOTE] FqnToConfig with regex-based FQN matching is supported in torchao >= 0.17. Older versions will silently leave the MoE expert weights in BF16.

Quick Start

Requirements

bash

pip install --extra-index-url https://download.pytorch.org/whl/cpu \
            --extra-index-url https://wheels.vllm.ai/cpu/ \
    torch==2.11.0+cpu \
    vllm==0.22.0 \
    torchao==0.17.0 \
    "lm-eval[vllm]==0.4.12" \
    huggingface_hub

CPU runtime libraries (only needed if not already present):

conda install -c conda-forge gperftools=2.17.2 llvm-openmp=18.1.8 --no-deps -y

Recommended environment variables

bash

# TorchInductor + zentorch
export TORCHINDUCTOR_FREEZING=1
export TORCHINDUCTOR_AUTOGRAD_CACHE=0
export VLLM_USE_AOT_COMPILE=0
export ZENDNNL_MATMUL_ALGO=1
export ZENTORCH_FUSED_MOE=1                    # required for gpt-oss-20b (MoE)

# Required CPU runtime libraries
export LD_PRELOAD="<path to lib>/libtcmalloc_minimal.so.4:<path to lib>/libiomp5.so${LD_PRELOAD:+:$LD_PRELOAD}"

Locate the libraries with find / -name 'libtcmalloc_minimal.so.4' and find / -name 'libiomp5.so', then substitute the resulting directory for <path to lib>.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	DA8W8 (this model)	Dynamic Quant Difference (baseline: BF16)
GSM8K (5-shot, exact-match flexible)	-	88.17	-

Evaluation Command

bash

lm_eval \
    --model vllm \
    --model_args pretrained=amd/gpt-oss-20b-da8w8-torchao-v0.17.0,tokenizer=unsloth/gpt-oss-20b-BF16,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --apply_chat_template \
    --output_path .

Limitations

Version Lock: This model is quantized with TorchAO v0.17.0 and is compatible only with PyTorch v2.11.0 / ZenDNN v6.0.0. It will not load correctly on other PyTorch versions.
MoE Expert Granularity: Expert weights are quantized at per-row granularity (rather than per-tensor) because the expert tensors are 3D ([num_experts, in, out]) parameters; per-tensor scales were observed to be too coarse across experts.
CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

amd/gpt-oss-20b-BF16-da8w8-torchao-v0.17.0

gpt-oss-20b-da8w8-torchao-v0.17.0

Model Overview

Quantization

Quick Start

Requirements

Recommended environment variables

Evaluation

Evaluation Command

Limitations

License

amd/
gpt-oss-20b-BF16-da8w8-torchao-v0.17.0