OPT: Optimized for AMD ROCm
OPT is Meta's causal language model family, served via
vLLM's OpenAI-compatible HTTP API. This repository packages evaluation/inference for causal language modeling using vLLM, exported and validated for
AMD ROCm so it runs efficiently on AMD GPUs.
This is based on the implementation of OPT found
here.
This repository contains configurations and scripts optimized for
AMD® ROCm™ platforms. You can use the
facebook-opt AMD scripts to reproduce results or export with custom configurations. More details on model performance can be found
here.
Task Overview
Task: Causal language modeling (text generation)
Dataset: HellaSwag, ARC-Challenge, Winogrande, MMLU, GSM8K (via lm-evaluation-harness)
Output metrics: acc / acc_norm / exact_match per task (not perplexity)
Model variants: Default is opt-1.3b (facebook/opt-1.3b). Override with MODEL_SIZE=125m|350m|1.3b|2.7b|6.7b. Only one variant's weights live on disk at a time; downloading another deletes the previous one's. All OPT variants are not gated on Hugging Face — no token or license acceptance required. All variants share OPT's 2048-token positional-embedding ceiling.
vLLM note: This version of vLLM does not work on kraken2e devices — ROCm has no supporting kernels for gfx1153.
AMD ROCm Optimization
This model export has been adapted and validated for AMD Instinct™ / Radeon™ GPUs running ROCm. Key points:
- Validated backends: vLLM (ROCm-enabled, OpenAI-compatible HTTP API).
- No code changes required versus the upstream OPT implementation — only environment/runtime configuration differs.
- Not supported on kraken2e devices (gfx1153) — ROCm lacks supporting kernels for this vLLM version.
| Runtime | Precision | Backend | Hardware | Notes |
|---|
| GPU | — | vLLM | AMD GPU (ROCm) | OpenAI-compatible HTTP API |
Getting Started
For setup instructions, evaluation scripts, and custom configuration options, see the
facebook-opt on GitHub.
Model Details
Model Type: Causal language model (decoder-only transformer)
Base Model: facebook/opt-1.3b (default) — opt-125m, opt-350m, opt-2.7b, opt-6.7b also supported
Model Stats:
- Model variant: opt-1.3b (default) — opt-125m, opt-350m, opt-2.7b, opt-6.7b also supported
- Positional-embedding ceiling: 2048 tokens (shared across all variants)
- Number of parameters:
1.3B (default variant; 125M/350M/2.7B/6.7B also available)
Accuracy Pipeline
Accuracy is measured with
EleutherAI's lm-evaluation-harness against the server's OpenAI-compatible
/v1/completions endpoint. Few-shot counts follow the Open LLM Leaderboard convention.
Metrics Explained
| Metric | Description |
|---|
| acc / acc_norm | Accuracy on multiple-choice tasks (HellaSwag, ARC-Challenge, Winogrande, MMLU); acc_norm length-normalizes answer choices. |
| exact_match | Exact-match accuracy for generative tasks (GSM8K). |
Accuracy Results
Measured results (LIMIT=100 smoke-test subset) — from the committed METRICS_SUMMARY.md, last updated 2026-05-11; run make eval-accuracy-all-slow and make metrics to refresh:
| Variant | Task | Metric | Value |
|---|
| opt-125m | arc_challenge | acc / acc_norm | 0.1900 / 0.2100 |
| opt-125m | hellaswag | acc / acc_norm | 0.3600 / 0.4200 |
| opt-125m | winogrande | acc | 0.5300 |
| opt-125m | mmlu | acc | 0.2616 |
| opt-125m | gsm8k | exact_match | 0.0000 |
| opt-350m | arc_challenge | acc / acc_norm | 0.1700 / 0.2200 |
| opt-350m | hellaswag | acc / acc_norm | 0.4100 / 0.4700 |
| opt-350m | winogrande | acc | 0.5800 |
opt-1.3b (default), opt-2.7b, and opt-6.7b had not yet been run as of this writing (TBD in the source README).
Dig Deeper
Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?
The GitHub repository includes:
- Setup and prerequisites for ROCm environments
- Scripts for benchmarking, model switching, and server management
- Additional model variants and datasets
- Benchmarking and reproduction instructions
