rag_docs/opt_overview.md

# OPT Overview

## What Is OPT?

OPT (Open Pre-trained Transformer) is a family of language models released by Meta in 2022. OPT was designed to replicate GPT-3's architecture and performance while being openly available to researchers. It uses a decoder-only transformer architecture similar to GPT-2 but with options for much larger sizes.

## Architecture Details

OPT's architecture is close to GPT-2 but has some differences:

| Property | OPT-125M | OPT-350M | OPT-1.3B |
|----------|----------|----------|----------|
| Parameters | 125M | 350M | 1.3B |
| Layers | 12 | 24 | 24 |
| Attention Heads | 12 | 16 | 32 |
| Hidden Dimension | 768 | 1024 | 2048 |
| Vocabulary Size | 50,272 | 50,272 | 50,272 |

### Key Differences from GPT-2

- **ReLU activation**: OPT uses **ReLU** instead of GPT-2's GELU. This is the only model in the dashboard with ReLU, making it useful for comparing how activation functions affect MLP behavior.
- **Learned positional embeddings**: Like GPT-2, OPT uses learned absolute position embeddings (unlike Pythia's or Qwen's RoPE)
- **LayerNorm placement**: OPT uses pre-norm LayerNorm (applied before each sublayer), which is slightly different from GPT-2's original arrangement
- **Larger variants available**: OPT scales up to 175 billion parameters, though only smaller variants are practical for interactive use

### Similarities to GPT-2

- Same general decoder-only architecture
- Same tokenizer style (BPE with ~50K vocabulary)
- Same attention mechanism (standard multi-head self-attention)
- Similar training objective (next-token prediction)

## What to Expect in the Dashboard

When using OPT models:

- **OPT-125M is very similar to GPT-2**: Same number of layers (12), heads (12), and hidden dimension (768). You'll see similar attention patterns and predictions.
- **Different module paths**: The dashboard auto-detects OPT's internal structure (e.g., `model.decoder.layers.N.self_attn`), so hooking works automatically.
- **Tokenization**: OPT's tokenizer is very similar to GPT-2's, so the same text usually produces similar (but not identical) token sequences.
- **Good for comparison**: Running the same prompt on GPT-2 and OPT-125M can show how similar architectures with different training data and activation functions produce different predictions.

## HuggingFace Model IDs

- `facebook/opt-125m` (in dropdown)
- `facebook/opt-350m`, `facebook/opt-1.3b` (larger, enter manually)