| ---
|
| library_name: pytorch
|
| pipeline_tag: text-generation
|
| tags:
|
| - custom-pytorch
|
| - gpt
|
| - causal-language-model
|
| ---
|
|
|
| # Pluto
|
|
|
| Pluto is a small custom GPT-style causal language model implemented in
|
| PyTorch. I was teaching myself how models are made. This repository contains an inference checkpoint and the source
|
| needed to load it.
|
|
|
| ## Usage
|
|
|
| The model uses the GPT-2 byte-pair encoding tokenizer from `tiktoken`. For
|
| instruction fine-tuning, format prompts as:
|
|
|
| ```text
|
| ### Instruction:
|
| Your instruction here
|
|
|
| ### Response:
|
| ```
|
|
|
| The checkpoint is a custom PyTorch checkpoint, not a Transformers
|
| `AutoModelForCausalLM` checkpoint. Load it with the `GPT` and `GPTConfig`
|
| classes from `model.py`.
|
|
|
| ## Limitations
|
|
|
| This is an experimental small model. Outputs may be inaccurate, repetitive,
|
| or unsafe. The context window is 256 tokens.
|
|
|
| ## Reproducibility
|
|
|
| The release tag records the checkpoint selected for this version. Training
|
| checkpoints and optimizer state should be retained privately or in a separate
|
| training archive rather than bundled with the inference release.
|
|
|
| Find the full blog post here: https://bennymach.com/blog/pluto/
|
|
|
