File size: 4,914 Bytes
54289f5 76982da 54289f5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | ---
library_name: vllm
license: other
tags:
- foundation
- amd
- rocm
- text-generation
pipeline_tag: text-generation
---
# OPT: Optimized for AMD ROCm
OPT is Meta's causal language model family, served via [vLLM](https://github.com/vllm-project/vllm)'s OpenAI-compatible HTTP API. This repository packages evaluation/inference for causal language modeling using vLLM, exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs.
This is based on the implementation of OPT found [here](https://huggingface.co/facebook/opt-1.3b).
This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [facebook-opt AMD scripts](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/facebook-opt) to reproduce results or export with custom configurations. More details on model performance can be found [here](#accuracy-pipeline).
---
## Task Overview
**Task:** Causal language modeling (text generation)
**Dataset:** HellaSwag, ARC-Challenge, Winogrande, MMLU, GSM8K (via `lm-evaluation-harness`)
**Output metrics:** `acc` / `acc_norm` / `exact_match` per task (not perplexity)
> **Model variants:** Default is **opt-1.3b** (`facebook/opt-1.3b`). Override with `MODEL_SIZE=125m|350m|1.3b|2.7b|6.7b`. Only one variant's weights live on disk at a time; downloading another deletes the previous one's. All OPT variants are not gated on Hugging Face — no token or license acceptance required. All variants share OPT's 2048-token positional-embedding ceiling.
> **vLLM note:** This version of vLLM does not work on kraken2e devices — ROCm has no supporting kernels for gfx1153.
---
## AMD ROCm Optimization
This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**. Key points:
- Validated backends: **vLLM** (ROCm-enabled, OpenAI-compatible HTTP API).
- No code changes required versus the upstream OPT implementation — only environment/runtime configuration differs.
- Not supported on kraken2e devices (gfx1153) — ROCm lacks supporting kernels for this vLLM version.
| Runtime | Precision | Backend | Hardware | Notes |
|---|---|---|---|---|
| GPU | — | vLLM | AMD GPU (ROCm) | OpenAI-compatible HTTP API |
---
## Getting Started
For setup instructions, evaluation scripts, and custom configuration options, see the [facebook-opt on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/facebook-opt).
---
## Model Details
**Model Type:** Causal language model (decoder-only transformer)
**Base Model:** `facebook/opt-1.3b` (default) — `opt-125m`, `opt-350m`, `opt-2.7b`, `opt-6.7b` also supported
**Model Stats:**
- Model variant: opt-1.3b (default) — opt-125m, opt-350m, opt-2.7b, opt-6.7b also supported
- Positional-embedding ceiling: 2048 tokens (shared across all variants)
- Number of parameters: `1.3B` (default variant; 125M/350M/2.7B/6.7B also available)
---
## Accuracy Pipeline
Accuracy is measured with [EleutherAI's lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) against the server's OpenAI-compatible `/v1/completions` endpoint. Few-shot counts follow the Open LLM Leaderboard convention.
### Metrics Explained
| Metric | Description |
|--------|-------------|
| acc / acc_norm | Accuracy on multiple-choice tasks (HellaSwag, ARC-Challenge, Winogrande, MMLU); `acc_norm` length-normalizes answer choices. |
| exact_match | Exact-match accuracy for generative tasks (GSM8K). |
### Accuracy Results
**Measured results (LIMIT=100 smoke-test subset)** — from the committed `METRICS_SUMMARY.md`, last updated 2026-05-11; run `make eval-accuracy-all-slow` and `make metrics` to refresh:
<!-- accuracy-table-start -->
| Variant | Task | Metric | Value |
|---------|------|--------|-------|
| opt-125m | arc_challenge | acc / acc_norm | 0.1900 / 0.2100 |
| opt-125m | hellaswag | acc / acc_norm | 0.3600 / 0.4200 |
| opt-125m | winogrande | acc | 0.5300 |
| opt-125m | mmlu | acc | 0.2616 |
| opt-125m | gsm8k | exact_match | 0.0000 |
| opt-350m | arc_challenge | acc / acc_norm | 0.1700 / 0.2200 |
| opt-350m | hellaswag | acc / acc_norm | 0.4100 / 0.4700 |
| opt-350m | winogrande | acc | 0.5800 |
<!-- accuracy-table-end -->
opt-1.3b (default), opt-2.7b, and opt-6.7b had not yet been run as of this writing (TBD in the source README).
---
## Dig Deeper
Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?
📂 **[View the full project on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/facebook-opt)**
The GitHub repository includes:
- Setup and prerequisites for ROCm environments
- Scripts for benchmarking, model switching, and server management
- Additional model variants and datasets
- Benchmarking and reproduction instructions
|