language: en
license: mit
tags:
- mechanistic-interpretability
- tiny-stories
- gpt-neo
- narrative-coherence
- research
datasets:
- roneneldan/TinyStories
base_model: EleutherAI/gpt-neo-125m
NSTS: Narrative Structure in Tiny Stories
Model checkpoints accompanying the paper "Fluency Is Not Coherence: What Small Language Models Actually Learn".
Overview
This repository contains a depth-controlled width series of GPT-Neo models trained from scratch on FK-filtered subsets of the TinyStories corpus, designed to study the dissociation between fluency and narrative coherence in small language models.
The central finding: fluency scales monotonically with model size and training duration; narrative coherence plateaus to a low ceiling (~2.4/5.0) that additional capacity, training, and corpus enrichment all fail to raise. This is not a capacity problem, not a data problem — it is a property of what next-token cross-entropy prediction selects for.
Model Series
All models use the GPT-Neo architecture with 8 layers, 8 heads, varying only hidden dimension (depth-controlled width series).
| Size | Hidden | Non-emb params | Total params |
|---|---|---|---|
| 1M | 64 | ~1.4M | ~3.6M |
| 5M | 144 | ~5.3M | ~9.2M |
| 10M | 256 | ~10.7M | ~19.3M |
| 28M | 480 | ~28.4M | ~46.5M |
| 33M | 512 | ~33.4M | ~51.0M |
Training Conditions
Two FK-filtered subsets of TinyStories, holding vocabulary distribution constant:
- Condition A: FK grade < 3 — 860K stories, mean sentence length 8.05 words
- Condition B: FK grade 4–5 — 574K stories, mean sentence length 12.35 words
Checkpoints saved every 100 optimiser steps (17 checkpoints for Cond A, 23 for Cond B).
Checkpoint Structure
nsts_cond{A|B}_{size}/
checkpoint-100/
checkpoint-200/
...
checkpoint-1680/ # Cond A final
checkpoint-2240/ # Cond B final
Note: nsts_condA_5M is named nsts_condA_5M_ep2 (2-epoch run); nsts_condB_5M is named nsts_condB_5M_ep4 (4-epoch run), but these Extended Epoch runs still contain checkpoints at 1680/2240 for direct comparison.
Loading a Checkpoint
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
# Download a specific checkpoint
path = snapshot_download(
repo_id="Dan44788/NSTS",
allow_patterns="nsts_condA_10M/checkpoint-1680/*"
)
model = AutoModelForCausalLM.from_pretrained(f"{path}/nsts_condA_10M/checkpoint-1680")
tokenizer = AutoTokenizer.from_pretrained(f"{path}/nsts_condA_10M/checkpoint-1680")
Key Results
- Fluency and coherence dissociate sharply at ~800–1000 training steps — a transition point that is scale-invariant across all model sizes
- Best coherence score achieved: 2.43/5.0 ("a story that resolves with gaps") at 28M parameters
- Seven direct interventions (corpus enrichment, structured narrative injection, causal markers, extended training, prefix prompting) all produced null results on coherence
- BLiMP probing reveals the same local/non-local split independently: local grammatical constraints scale with width; non-local constraints are flat at chance
Intended Use
These checkpoints are intended as a test-bed for mechanistic interpretability research at tractable scale. The clean behavioural dissociation between fluency and coherence provides a well-characterised target for circuit-level analysis.
Weight trajectories
Citation
Fluency Is Not Coherence: What Small Language Models Actually Learn (v1)