---
language:
- en
license: apache-2.0
tags:
- pytorch
- gpt
- from-scratch
- small-lm
- fitness
pipeline_tag: text-generation
---

# Koachie 312M — a language model built from scratch for £60

Koachie is a 312M-parameter GPT-style language model built entirely from
scratch by [Baller Knowledge](https://koachiefitai.co.uk) in Glasgow, Scotland
— own tokenizer, own architecture implementation, own training loop. It is not
a fine-tune of any existing model and no external model's weights were used at
any point.

Total pretraining cost was roughly **£60 of rented GPU time**. It serves real
users today at [koachiefitai.co.uk](https://koachiefitai.co.uk).

## Architecture

| | |
|---|---|
| Parameters | 312.06M |
| Layers / width / heads | 24 × 1024d × 16h (head_dim 64) |
| Positions | RoPE |
| Norm / MLP | RMSNorm (pre-norm) / SwiGLU |
| Attention | fused flash attention (PyTorch SDPA) |
| Context | 512 tokens |
| Vocabulary | 8,000 (custom BPE, trained from the corpus) |
| Embeddings | tied |

## Training

- **Data:** ~40B tokens, predominantly FineWeb-Edu, with Wikipedia, Project
  Gutenberg, PubMed Central open-access and OASST/Dolly dialogue. Licence-aware
  curation throughout (Reddit-derived data deliberately excluded).
- **Optimiser:** Muon (Newton–Schulz orthogonalised momentum) for hidden
  matrices, AdamW for embeddings/norms; WSD (warmup–stable–decay) schedule.
- **Hardware:** single-GPU rentals (RTX 5090 class) plus a desktop RTX 3070;
  DDP path written and measured at 137k tokens/sec on 4× RTX 5090.
- **A finding worth sharing:** Muon learning-rate stability followed a
  reproducible law across two machines — each halving of LR extended the
  stable training horizon roughly 8× (0.006 diverged near 2k steps, 0.003
  near 25k, 0.0015 near 40k, 0.00075 past 180k).

## Evaluation

EleutherAI lm-evaluation-harness, **full test sets, no sampling**. Two
checkpoints: the pretrained base and the conversational SFT that serves the
live product.

| task (acc, %) | base | SFT |
|---|---|---|
| SciQ | **71.9** | 66.2 |
| PIQA | 62.0 | 61.7 |
| WinoGrande | 50.4 | 51.4 |
| ARC-Easy | 45.7 | 45.2 |
| OpenBookQA (norm) | 31.8 | 31.8 |
| HellaSwag (norm) | 29.4 | 29.6 |
| ARC-Challenge (norm) | 22.9 | 25.3 |
| LAMBADA (ppl ↓) | **176.1** | 241.0 |

Roughly GPT-2-class performance. The base is the stronger language model
(note the perplexity gap); the SFT trades some of that for conversational
behaviour, which is the expected pattern.

## Honest limitations

This is a small model and it behaves like one. Closed-book, it will confidently
state wrong facts; ARC-Challenge is near chance. The production system around
it does not rely on the model's memory: factual questions are resolved against
Wikidata's structured records or answered by quoting a retrieved source
verbatim, generated answers are marked by a judge before they ship, and
medical red flags are intercepted and referred to a professional. The model is
the voice; verification is the product.

## Files

- `koachie-312m-base.pt` — pretrained base checkpoint (step 304,000, ~40B tokens)
- `tokenizer.json` — the 8k custom BPE tokenizer
- `lm_eval_base.json` / `lm_eval_sft.json` — raw harness output for the table above

Weights are a plain PyTorch `state_dict` under `model` with `model_config`
alongside; the reference implementation (model code, training loop, evaluation
adapters) lives in the project repository.

## Intended use

Research into small-model training economics, retrieval-first product
architectures, and as evidence that a working language model is buildable at
consumer cost. Not a general-purpose assistant on its own.

Built by Steven McKinney, Baller Knowledge UK Ltd, Glasgow.