File size: 1,177 Bytes
28fc50d bd92a0b 28fc50d b544b42 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | ---
library_name: pytorch
pipeline_tag: text-generation
tags:
- custom-pytorch
- gpt
- causal-language-model
---
# Pluto
Pluto is a small custom GPT-style causal language model implemented in
PyTorch. I was teaching myself how models are made. This repository contains an inference checkpoint and the source
needed to load it.
## Usage
The model uses the GPT-2 byte-pair encoding tokenizer from `tiktoken`. For
instruction fine-tuning, format prompts as:
```text
### Instruction:
Your instruction here
### Response:
```
The checkpoint is a custom PyTorch checkpoint, not a Transformers
`AutoModelForCausalLM` checkpoint. Load it with the `GPT` and `GPTConfig`
classes from `model.py`.
## Limitations
This is an experimental small model. Outputs may be inaccurate, repetitive,
or unsafe. The context window is 256 tokens.
## Reproducibility
The release tag records the checkpoint selected for this version. Training
checkpoints and optimizer state should be retained privately or in a separate
training archive rather than bundled with the inference release.
Find the full blog post here: https://bennymach.com/blog/pluto/
|