You can deploy this model and start earning money today!

amd/
Phi-4-reasoning-plus-w4a16-llmcompressor-v0.12.0

Name: Phi-4-reasoning-plus-w4a16-llmcompressor-v0.12.0
Availability: InStock

Views

No views yet

Phi-4-reasoning-plus-w4a16-llmcompressor-v0.12.0

Model Overview

Model Architecture: Phi3ForCausalLM
- Input: Text
- Output: Text
Source Model: Phi-4-reasoning-plus
Supported Hardware: AMD EPYC (CPU inference)
Preferred Operating System: Linux
Inference Engine: vLLM v0.26.0
Quantization Framework: LLM Compressor v0.12.0
Quantization Method: 4-bit Weight-Only Quantization (W4A16)
Compatible Stack:
- ZenDNN v6.1.0
- ZenTorch v2.11.0.3
- PyTorch v2.11.0
- LLM Compressor v0.12.0
- vLLM v0.26.0

This is a quantized version of Phi-4-reasoning-plus created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Phi-4-reasoning-plus using LLM Compressor via the GPTQ algorithm.

Method: 4-bit Weight-Only Quantization (W4A16)
Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128
Weights: INT4, symmetric, group-wise (group_size=128, actorder=static), stored as pack-quantized
Activations: BF16 (unquantized)
Ignored modules: lm_head
Calibration: 128 examples from HuggingFaceH4/ultrachat_200k at a max sequence length of 2048

python

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier

model_id = "microsoft/Phi-4-reasoning-plus"
output_dir = "./Phi-4-reasoning-plus-w4a16-llmcompressor-v0.12.0"

NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 2048

# Step 1: Load the BF16 model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Load calibration data. GPTQ is data-driven: it needs real activations
# to build the per-layer Hessians used to compensate the rounding error.
ds = load_dataset(
    "HuggingFaceH4/ultrachat_200k",
    split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]",
)
ds = ds.map(
    lambda example: {"text": "\n".join(m["content"] for m in example["messages"])},
    remove_columns=ds.column_names,
)

# Step 3: Define the W4A16 GPTQ recipe
recipe = GPTQModifier(scheme="W4A16", targets="Linear", ignore=["lm_head"])

# Step 4: One-shot quantize with calibration and save in compressed-tensors format
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

python

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Phi-4-reasoning-plus-w4a16-llmcompressor-v0.12.0",
    dtype="bfloat16",
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.11.0
zentorch==2.11.0.3
vllm==0.26.0
llmcompressor==0.12.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

bash

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	W4A16 (this model)	Recovery
GSM8K (5-shot)	0.8704	0.8688	99.82%

Evaluation Command

bash

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Phi-4-reasoning-plus-w4a16-llmcompressor-v0.12.0,dtype=bfloat16 \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.11.0.3 / PyTorch v2.11.0. It may not load correctly on other versions.
CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
Accuracy Trade-off: 4-bit weight-only quantization is more aggressive than INT8. This checkpoint recovers 99.82% of the BF16 GSM8K score.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.