library_name: pytorch
pipeline_tag: text-generation
tags:
- custom-pytorch
- gpt
- causal-language-model
Pluto
Pluto is a small custom GPT-style causal language model implemented in PyTorch. I was teaching myself how models are made. This repository contains an inference checkpoint and the source needed to load it.
Usage
The model uses the GPT-2 byte-pair encoding tokenizer from tiktoken. For
instruction fine-tuning, format prompts as:
### Instruction:
Your instruction here
### Response:
The checkpoint is a custom PyTorch checkpoint, not a Transformers
AutoModelForCausalLM checkpoint. Load it with the GPT and GPTConfig
classes from model.py.
Limitations
This is an experimental small model. Outputs may be inaccurate, repetitive, or unsafe. The context window is 256 tokens.
Reproducibility
The release tag records the checkpoint selected for this version. Training checkpoints and optimizer state should be retained privately or in a separate training archive rather than bundled with the inference release.
Find the full blog post here: https://bennymach.com/blog/pluto/