---
license: apache-2.0
language:
- en
library_name: pytorch
---

# theOG-50M

An experimental PyTorch checkpoint from a small language model research
project (compute-elastic 50M-class pilot).

## Files

- `model_state.pt` — raw state dict (`torch.load(..., weights_only=True)`, fp32, ~208 MB).
- `config.json` — architecture: 16000 vocab, hidden 512, 10 layers, 4 heads x
  head_dim 128, SwiGLU, RoPE + learned positional embeddings, tied embeddings,
  MoE with 2 experts (top_k=1).
- `tokenizer.json` — BPE tokenizer (15853 tokens) matching the checkpoint.
  Reserved ids: `<|think|>` = 15851, `<|/think|>` = 15852.

## Size

Effective model size is **~50M active parameters** (50.7M per the
`pilot_50m_deep` config: MoE with 2 experts, only 1 expert active per token,
embeddings tied). The raw `.pt` state dict totals 60.2M values because it
counts the tied embedding twice and includes both dormant experts.

## Load and run

```python
import torch
from tokenizers import Tokenizer

sd = torch.load("model_state.pt", map_location="cpu", weights_only=True)
tok = Tokenizer.from_file("tokenizer.json")
```

No reference implementation is included in this repo, so the exact wiring must
be reconstructed from the tensor shapes in `config.json` + `model_state.pt`.

## HumanEval (50-task subset, execution-graded)

Measured on the project's canonical HumanEval harness: sample k completions
(temp 0.8, top-p 0.95, max 220 new tokens), execution-select the first passing
one. Pass iff `prompt + completion + test + check(entry_point)` exits 0.

| protocol | pass@1 | pass@k |
|----------|--------|--------|
| k=8  | 11/50  | 16/50 |
| k=16 | 11/50  | 19/50 |
| k=32 | 11/50  | 20/50 |
| CodeGen-350M greedy (reference) | 15/50 | 15/50 |

The full per-task completions and grader are in the project repo
(`scripts/bench_humaneval.py`, `evaluation/results/humaneval_og50_k{8,16,32}.json`).

## Status

Experimental checkpoint from a research run. The numbers above are the
measured ones with the protocol stated; single-sample pass@1 is significant (11/50),
sampling with execution selection brings it to ~16-20/50.

## Safety note

The state dict was disassembled before loading; it contains only standard
tensor-rebuild globals and is safe to load with `weights_only=True`.