Property	Value
Method	FP8 (W8A8)
Tool	llmcompressor
Calibration Dataset	`open_platypus`
Calibration Samples	128
Max Sequence Length	1024
Targets	All `Linear` layers
Ignored Layers	`lm_head`, `gate` (MoE router)

The MoE routing gate layers are kept in full precision to preserve expert-selection accuracy. The lm_head is kept in full precision following standard practice.

Model Size

	Original (BF16)	This Model (FP8)
Disk Size	~94 GB	~47 GB
Shards	19	10
Reduction	—	~50%

Usage

python

import os
os.environ["FORCE_DISABLE_VISION"] = "1"

from google.colab import userdata
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

HF_TOKEN = userdata.get('HF_TOKEN')
USERNAME = "frankmorales2020"
REPO_ID = f"{USERNAME}/Mixtral-8x7B-Instruct-v0.1-fp8"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID, token=HF_TOKEN)

# Fix: explicitly set pad_token to eos_token
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"   # left-pad for decoder-only models

model = AutoModelForCausalLM.from_pretrained(
    REPO_ID,
    token=HF_TOKEN,
    device_map="auto",
    dtype=torch.bfloat16,
)

# Fix: set pad_token_id on the model config too
model.config.pad_token_id = tokenizer.eos_token_id

def run_inference(prompt):
    inputs = tokenizer(
        f"[INST] {prompt} [/INST]",
        return_tensors="pt",
        padding=True,
    ).to(model.device)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            temperature=0.7,
            top_p=0.9,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,  # silences the warning
        )

    return tokenizer.decode(
        outputs[0][inputs["input_ids"].shape[1]:],
        skip_special_tokens=True
    ).strip()

prompts = [
    "Explain mixture of experts architecture in simple terms.",
    "Write a Python function to compute the Fibonacci sequence.",
    "What are the advantages of FP8 quantization over INT8?",
]

for prompt in prompts:
    print(f"\n{'='*60}")
    print(f"PROMPT: {prompt}")
    print(f"{'='*60}")
    print(f"RESPONSE:\n{run_inference(prompt)}")
    print()

Note: FP8 inference requires an NVIDIA GPU with Ada Lovelace (RTX 4000 series), Hopper (H100), or newer architecture. Ampere (A100) supports FP8 storage but not native FP8 compute.

Hardware Requirements

Component	Minimum
GPU VRAM	24 GB (single GPU, e.g. A10G / RTX 4090)
GPU Architecture	Ada Lovelace / Hopper or newer for native FP8
CUDA	12.8+

Base Model

Base: mistralai/Mixtral-8x7B-Instruct-v0.1
Architecture: Mixtral MoE — 8 experts, top-2 routing, 46.7B total / ~13B active parameters
License: Apache 2.0

Limitations

Slight accuracy degradation vs. the BF16 base (typical for FP8 PTQ)
FP8 compute not available on Volta, Turing, or Ampere GPUs
Not evaluated on benchmarks post-quantization — use with appropriate validation for production workloads

frankmorales2020/Mixtral-8x7B-Instruct-v0.1-fp8

Mixtral-8x7B-Instruct-v0.1 — FP8 Quantized

Quantization Details

Model Size

Usage

Hardware Requirements

Base Model

Limitations

frankmorales2020/
Mixtral-8x7B-Instruct-v0.1-fp8