| --- |
| library_name: vllm |
| license: other |
| tags: |
| - foundation |
| - amd |
| - rocm |
| - text-generation |
| pipeline_tag: text-generation |
| --- |
| |
|  |
|
|
| # OPT: Optimized for AMD ROCm |
|
|
| OPT is Meta's causal language model family, served via [vLLM](https://github.com/vllm-project/vllm)'s OpenAI-compatible HTTP API. This repository packages evaluation/inference for causal language modeling using vLLM, exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs. |
|
|
| This is based on the implementation of OPT found [here](https://huggingface.co/facebook/opt-1.3b). |
| This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [facebook-opt AMD scripts](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/facebook-opt) to reproduce results or export with custom configurations. More details on model performance can be found [here](#accuracy-pipeline). |
|
|
| --- |
|
|
| ## Task Overview |
|
|
| **Task:** Causal language modeling (text generation) |
|
|
| **Dataset:** HellaSwag, ARC-Challenge, Winogrande, MMLU, GSM8K (via `lm-evaluation-harness`) |
|
|
| **Output metrics:** `acc` / `acc_norm` / `exact_match` per task (not perplexity) |
|
|
| > **Model variants:** Default is **opt-1.3b** (`facebook/opt-1.3b`). Override with `MODEL_SIZE=125m|350m|1.3b|2.7b|6.7b`. Only one variant's weights live on disk at a time; downloading another deletes the previous one's. All OPT variants are not gated on Hugging Face — no token or license acceptance required. All variants share OPT's 2048-token positional-embedding ceiling. |
| |
| > **vLLM note:** This version of vLLM does not work on kraken2e devices — ROCm has no supporting kernels for gfx1153. |
| |
| --- |
| |
| ## AMD ROCm Optimization |
| |
| This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**. Key points: |
| |
| - Validated backends: **vLLM** (ROCm-enabled, OpenAI-compatible HTTP API). |
| - No code changes required versus the upstream OPT implementation — only environment/runtime configuration differs. |
| - Not supported on kraken2e devices (gfx1153) — ROCm lacks supporting kernels for this vLLM version. |
| |
| | Runtime | Precision | Backend | Hardware | Notes | |
| |---|---|---|---|---| |
| | GPU | — | vLLM | AMD GPU (ROCm) | OpenAI-compatible HTTP API | |
| |
| --- |
| |
| ## Getting Started |
| |
| For setup instructions, evaluation scripts, and custom configuration options, see the [facebook-opt on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/facebook-opt). |
| |
| --- |
| |
| ## Model Details |
| |
| **Model Type:** Causal language model (decoder-only transformer) |
| |
| **Base Model:** `facebook/opt-1.3b` (default) — `opt-125m`, `opt-350m`, `opt-2.7b`, `opt-6.7b` also supported |
| |
| **Model Stats:** |
| - Model variant: opt-1.3b (default) — opt-125m, opt-350m, opt-2.7b, opt-6.7b also supported |
| - Positional-embedding ceiling: 2048 tokens (shared across all variants) |
| - Number of parameters: `1.3B` (default variant; 125M/350M/2.7B/6.7B also available) |
| |
| --- |
| |
| ## Accuracy Pipeline |
| |
| Accuracy is measured with [EleutherAI's lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) against the server's OpenAI-compatible `/v1/completions` endpoint. Few-shot counts follow the Open LLM Leaderboard convention. |
| |
| ### Metrics Explained |
| |
| | Metric | Description | |
| |--------|-------------| |
| | acc / acc_norm | Accuracy on multiple-choice tasks (HellaSwag, ARC-Challenge, Winogrande, MMLU); `acc_norm` length-normalizes answer choices. | |
| | exact_match | Exact-match accuracy for generative tasks (GSM8K). | |
|
|
| ### Accuracy Results |
|
|
| **Measured results (LIMIT=100 smoke-test subset)** — from the committed `METRICS_SUMMARY.md`, last updated 2026-05-11; run `make eval-accuracy-all-slow` and `make metrics` to refresh: |
|
|
| <!-- accuracy-table-start --> |
| | Variant | Task | Metric | Value | |
| |---------|------|--------|-------| |
| | opt-125m | arc_challenge | acc / acc_norm | 0.1900 / 0.2100 | |
| | opt-125m | hellaswag | acc / acc_norm | 0.3600 / 0.4200 | |
| | opt-125m | winogrande | acc | 0.5300 | |
| | opt-125m | mmlu | acc | 0.2616 | |
| | opt-125m | gsm8k | exact_match | 0.0000 | |
| | opt-350m | arc_challenge | acc / acc_norm | 0.1700 / 0.2200 | |
| | opt-350m | hellaswag | acc / acc_norm | 0.4100 / 0.4700 | |
| | opt-350m | winogrande | acc | 0.5800 | |
| <!-- accuracy-table-end --> |
| |
| opt-1.3b (default), opt-2.7b, and opt-6.7b had not yet been run as of this writing (TBD in the source README). |
| |
| --- |
| |
| ## Dig Deeper |
| |
| Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples? |
| |
| 📂 **[View the full project on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/facebook-opt)** |
| |
| The GitHub repository includes: |
| - Setup and prerequisites for ROCm environments |
| - Scripts for benchmarking, model switching, and server management |
| - Additional model variants and datasets |
| - Benchmarking and reproduction instructions |
| |
