Update image references to HF URLs

76982da verified about 14 hours ago

4.91 kB

library_name: vllm
license: other
tags:
  - foundation
  - amd
  - rocm
  - text-generation
pipeline_tag: text-generation

OPT: Optimized for AMD ROCm

OPT is Meta's causal language model family, served via vLLM's OpenAI-compatible HTTP API. This repository packages evaluation/inference for causal language modeling using vLLM, exported and validated for AMD ROCm so it runs efficiently on AMD GPUs.

This is based on the implementation of OPT found here. This repository contains configurations and scripts optimized for AMD® ROCm™ platforms. You can use the facebook-opt AMD scripts to reproduce results or export with custom configurations. More details on model performance can be found here.

Task Overview

Task: Causal language modeling (text generation)

Dataset: HellaSwag, ARC-Challenge, Winogrande, MMLU, GSM8K (via lm-evaluation-harness)

Output metrics: acc / acc_norm / exact_match per task (not perplexity)

Model variants: Default is opt-1.3b (facebook/opt-1.3b). Override with MODEL_SIZE=125m|350m|1.3b|2.7b|6.7b. Only one variant's weights live on disk at a time; downloading another deletes the previous one's. All OPT variants are not gated on Hugging Face — no token or license acceptance required. All variants share OPT's 2048-token positional-embedding ceiling.

vLLM note: This version of vLLM does not work on kraken2e devices — ROCm has no supporting kernels for gfx1153.

AMD ROCm Optimization

This model export has been adapted and validated for AMD Instinct™ / Radeon™ GPUs running ROCm. Key points:

Validated backends: vLLM (ROCm-enabled, OpenAI-compatible HTTP API).
No code changes required versus the upstream OPT implementation — only environment/runtime configuration differs.
Not supported on kraken2e devices (gfx1153) — ROCm lacks supporting kernels for this vLLM version.

Runtime	Precision	Backend	Hardware	Notes
GPU	—	vLLM	AMD GPU (ROCm)	OpenAI-compatible HTTP API

Getting Started

For setup instructions, evaluation scripts, and custom configuration options, see the facebook-opt on GitHub.

Model Details

Model Type: Causal language model (decoder-only transformer)

Base Model: facebook/opt-1.3b (default) — opt-125m, opt-350m, opt-2.7b, opt-6.7b also supported

Model Stats:

Model variant: opt-1.3b (default) — opt-125m, opt-350m, opt-2.7b, opt-6.7b also supported
Positional-embedding ceiling: 2048 tokens (shared across all variants)
Number of parameters: 1.3B (default variant; 125M/350M/2.7B/6.7B also available)

Accuracy Pipeline

Accuracy is measured with EleutherAI's lm-evaluation-harness against the server's OpenAI-compatible /v1/completions endpoint. Few-shot counts follow the Open LLM Leaderboard convention.

Metrics Explained

Metric	Description
acc / acc_norm	Accuracy on multiple-choice tasks (HellaSwag, ARC-Challenge, Winogrande, MMLU); `acc_norm` length-normalizes answer choices.
exact_match	Exact-match accuracy for generative tasks (GSM8K).

Accuracy Results

Measured results (LIMIT=100 smoke-test subset) — from the committed METRICS_SUMMARY.md, last updated 2026-05-11; run make eval-accuracy-all-slow and make metrics to refresh:

Variant	Task	Metric	Value
opt-125m	arc_challenge	acc / acc_norm	0.1900 / 0.2100
opt-125m	hellaswag	acc / acc_norm	0.3600 / 0.4200
opt-125m	winogrande	acc	0.5300
opt-125m	mmlu	acc	0.2616
opt-125m	gsm8k	exact_match	0.0000
opt-350m	arc_challenge	acc / acc_norm	0.1700 / 0.2200
opt-350m	hellaswag	acc / acc_norm	0.4100 / 0.4700
opt-350m	winogrande	acc	0.5800

opt-1.3b (default), opt-2.7b, and opt-6.7b had not yet been run as of this writing (TBD in the source README).

Dig Deeper

Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?

📂 View the full project on GitHub

The GitHub repository includes:

Setup and prerequisites for ROCm environments
Scripts for benchmarking, model switching, and server management
Additional model variants and datasets
Benchmarking and reproduction instructions