connaaa/interpgpt-standard-23M

InterpGPT — Standard Model (23M)

Part of the InterpGPT matched-pair release. This is the standard model; its counterpart is connaaa/interpgpt-adhd-23M. Both models share identical architecture and training recipe; only the training data distribution differs.

	Value
Parameters	23,471,104
Layers	6
Heads	8
d_model	512
d_head	64
d_mlp (SwiGLU)	1408
Vocab	8192 (custom BPE)
Context length	512
Norm	RMSNorm (ε = 1e-6)
Position	RoPE (half-half, base 10,000)
Activation	SwiGLU
Biases	none
Tied input/output embeddings	yes
Training tokens	~25k steps on task-decomposition corpus

What is this model for?

Given a task prompt, the model writes a step-by-step decomposition. The standard variant was trained on normal task decompositions (tasks → subtasks in straightforward order). The ADHD counterpart was trained on decompositions with smaller steps and interleaved micro-regulation actions (e.g. "sip water", "deep breath", "quick stretch").

The pair is the subject of a mechanistic-interpretability study. Phase 1 headline findings:

• Structural head-position swap. A step-layout-broadcast head lives at L3H0 in the standard model and at L3H5 in the ADHD model. Cross-model per-position attention profile cosine similarity is 0.997 at the matched (different-index) pair vs a same-index baseline of 0.66.
• Block-2 content circuit. P(regulation token) at step-onset positions jumps 17× between layer 1 and layer 2 in the ADHD model (0.014 → 0.251); the standard model never crosses 1% at any layer.
• High-specificity null-steering SAE feature. See the companion SAE repo connaaa/interpgpt-sae-phase5.

Input format

<|task|>Clean the kitchen<|steps|>Step 1 text<|sep|>Step 2 text<|sep|>...<|end|>

Loading

HuggingFace Transformers (custom code)

from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained(
    "connaaa/interpgpt-standard-23M", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "connaaa/interpgpt-standard-23M"
)

TransformerLens (recommended for interpretability)

The repo ships a TransformerLens-compatible bundle at hooked_transformer.pt:

from huggingface_hub import hf_hub_download
from transformer_lens import HookedTransformer, HookedTransformerConfig
import torch

path = hf_hub_download(
    "connaaa/interpgpt-standard-23M", "hooked_transformer.pt"
)
blob = torch.load(path, map_location="cpu", weights_only=False)
cfg_keep = {
    k: v for k, v in blob["config"].items()
    if k in HookedTransformerConfig.__dataclass_fields__ and not (
        isinstance(v, str) and v.startswith("torch.")
    )
}
cfg = HookedTransformerConfig(**cfg_keep)
model = HookedTransformer(cfg)
model.load_state_dict(blob["model_state_dict"])
model.eval()

Raw PyTorch / original TaskGPT class

# Pairs with gpt_model.py from https://github.com/cwklurks/interpgpt
from huggingface_hub import hf_hub_download
from gpt_model import GPTConfig, TaskGPT
import torch

path = hf_hub_download(
    "connaaa/interpgpt-standard-23M", "pytorch_model.pt"
)
blob = torch.load(path, map_location="cpu", weights_only=False)
model = TaskGPT(GPTConfig(**blob["config"]))
model.load_state_dict(blob["model_state_dict"])

Reproduce the head-swap finding

Open the companion Colab: notebooks/InterpGPT_HeadSwap.ipynb at github.com/cwklurks/interpgpt. End-to-end run on Colab free tier reproduces the 0.997 vs 0.66 comparison in under 15 minutes.

Training data

Custom task-decomposition corpus, two variants (standard vs ADHD) generated with the same task pool. Detailed dataset notes + generation scripts live in the main repo (preprocess.py, merge_data.py, rebuild_data.py, fix_adhd_data.py, shorten_adhd_steps.py).

License

MIT.

Intended use

Interpretability research. The model is intentionally small and domain-specific; not intended as a general-purpose chatbot.

Citation

@misc{interpgpt2026,
  title  = {{InterpGPT}: A matched-pair interpretability study of task-decomposition models},
  author = {Klann, Connor},
  year   = {2026},
  url    = {https://github.com/cwklurks/interpgpt}
}