---
library_name: pytorch
pipeline_tag: text-generation
tags:
- custom-pytorch
- gpt
- causal-language-model
---

# Pluto

Pluto is a small custom GPT-style causal language model implemented in
PyTorch. I was teaching myself how models are made. This repository contains an inference checkpoint and the source
needed to load it.

## Usage

The model uses the GPT-2 byte-pair encoding tokenizer from `tiktoken`. For
instruction fine-tuning, format prompts as:

```text
### Instruction:
Your instruction here

### Response:
```

The checkpoint is a custom PyTorch checkpoint, not a Transformers
`AutoModelForCausalLM` checkpoint. Load it with the `GPT` and `GPTConfig`
classes from `model.py`.

## Limitations

This is an experimental small model. Outputs may be inaccurate, repetitive,
or unsafe. The context window is 256 tokens.

## Reproducibility

The release tag records the checkpoint selected for this version. Training
checkpoints and optimizer state should be retained privately or in a separate
training archive rather than bundled with the inference release.

Find the full blog post here: https://bennymach.com/blog/pluto/