Pluto

Pluto is a small custom GPT-style causal language model implemented in PyTorch. I was teaching myself how models are made. This repository contains an inference checkpoint and the source needed to load it.

Usage

The model uses the GPT-2 byte-pair encoding tokenizer from tiktoken. For instruction fine-tuning, format prompts as:

### Instruction:
Your instruction here

### Response:

The checkpoint is a custom PyTorch checkpoint, not a Transformers AutoModelForCausalLM checkpoint. Load it with the GPT and GPTConfig classes from model.py.

Limitations

This is an experimental small model. Outputs may be inaccurate, repetitive, or unsafe. The context window is 256 tokens.

Reproducibility

The release tag records the checkpoint selected for this version. Training checkpoints and optimizer state should be retained privately or in a separate training archive rather than bundled with the inference release.

Find the full blog post here: https://bennymach.com/blog/pluto/