Update README.md
Browse files
README.md
CHANGED
|
@@ -2,7 +2,6 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
-
library_name: transformers
|
| 6 |
pipeline_tag: text-generation
|
| 7 |
tags:
|
| 8 |
- gpt2
|
|
@@ -18,6 +17,8 @@ A small GPT-2-style language model (~18.3M parameters) trained **completely from
|
|
| 18 |
|
| 19 |
This is a **text-completion model**, not an instruction-tuned chatbot. It's good at continuing short prose and simple stories. It is **not** good at answering questions, following instructions, or factual recall.
|
| 20 |
|
|
|
|
|
|
|
| 21 |
## Model details
|
| 22 |
|
| 23 |
| | |
|
|
@@ -37,33 +38,51 @@ This is a **text-completion model**, not an instruction-tuned chatbot. It's good
|
|
| 37 |
|
| 38 |
Trained primarily on **TinyStories** (~90M tokens) with a small amount of Alpaca-style data. The model learned general English sentence structure and simple narrative flow, not world knowledge.
|
| 39 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
## Intended use
|
| 41 |
|
| 42 |
- Story / prose continuation
|
| 43 |
- Experimentation and education (a clean, fully-from-scratch small LM)
|
| 44 |
|
| 45 |
-
## Limitations
|
| 46 |
-
|
| 47 |
-
- Completion only — will not reliably answer questions or follow instructions.
|
| 48 |
-
- No factual reliability; it will confidently make things up.
|
| 49 |
-
- Small context (256 tokens) and small vocab.
|
| 50 |
-
- English only.
|
| 51 |
-
|
| 52 |
## How to use
|
| 53 |
|
| 54 |
-
``
|
| 55 |
-
from transformers import AutoTokenizer, AutoModelForCausalLM
|
| 56 |
-
import torch
|
| 57 |
|
| 58 |
-
|
| 59 |
-
|
|
|
|
| 60 |
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
|
|
|
|
|
|
|
|
|
| 65 |
```
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
## License
|
| 68 |
|
| 69 |
Apache 2.0.
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
- en
|
|
|
|
| 5 |
pipeline_tag: text-generation
|
| 6 |
tags:
|
| 7 |
- gpt2
|
|
|
|
| 17 |
|
| 18 |
This is a **text-completion model**, not an instruction-tuned chatbot. It's good at continuing short prose and simple stories. It is **not** good at answering questions, following instructions, or factual recall.
|
| 19 |
|
| 20 |
+
> **Note:** this is a custom-format model, not a `transformers` model. You load it with the included scripts (`gpt2.py` + `chat.py`), not `AutoModelForCausalLM`.
|
| 21 |
+
|
| 22 |
## Model details
|
| 23 |
|
| 24 |
| | |
|
|
|
|
| 38 |
|
| 39 |
Trained primarily on **TinyStories** (~90M tokens) with a small amount of Alpaca-style data. The model learned general English sentence structure and simple narrative flow, not world knowledge.
|
| 40 |
|
| 41 |
+
## Files in this repo
|
| 42 |
+
|
| 43 |
+
| File | What it is |
|
| 44 |
+
|---|---|
|
| 45 |
+
| `model.safetensors` | The model weights |
|
| 46 |
+
| `config.json` | Architecture config (custom format) |
|
| 47 |
+
| `gpt2.py` | Model definition (the GPT-2-style architecture) |
|
| 48 |
+
| `chat.py` | Run / generate from the model |
|
| 49 |
+
| `tokenizer_bpe/vocab.json`, `tokenizer_bpe/merges.txt` | Byte-level BPE tokenizer |
|
| 50 |
+
| `big.pt` | Full training checkpoint (model + optimizer), for **resuming training** only |
|
| 51 |
+
| `train_big.py`, `prep_bpe.py` | Training and data-prep scripts |
|
| 52 |
+
|
| 53 |
## Intended use
|
| 54 |
|
| 55 |
- Story / prose continuation
|
| 56 |
- Experimentation and education (a clean, fully-from-scratch small LM)
|
| 57 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
## How to use
|
| 59 |
|
| 60 |
+
This model uses its own minimal code, not the `transformers` library.
|
|
|
|
|
|
|
| 61 |
|
| 62 |
+
```bash
|
| 63 |
+
# 1. Install deps
|
| 64 |
+
pip install torch tokenizers safetensors numpy
|
| 65 |
|
| 66 |
+
# 2. Download this repo (gives you the scripts + weights + tokenizer)
|
| 67 |
+
pip install huggingface_hub
|
| 68 |
+
hf download Gentraxyz/RecursiveComplete --local-dir RecursiveComplete
|
| 69 |
+
cd RecursiveComplete
|
| 70 |
+
|
| 71 |
+
# 3. Generate
|
| 72 |
+
python chat.py
|
| 73 |
```
|
| 74 |
|
| 75 |
+
`chat.py` loads `gpt2.py` (the architecture), the weights from `model.safetensors`, and the BPE tokenizer in `tokenizer_bpe/`, then lets you prompt the model for completions.
|
| 76 |
+
|
| 77 |
+
Tip: it's a completion model — give it the *start* of something ("Once upon a time there was a small robot who") rather than a question.
|
| 78 |
+
|
| 79 |
+
## Limitations
|
| 80 |
+
|
| 81 |
+
- Completion only — will not reliably answer questions or follow instructions.
|
| 82 |
+
- No factual reliability; it will confidently make things up.
|
| 83 |
+
- Small context (256 tokens) and small vocab (8192).
|
| 84 |
+
- English only.
|
| 85 |
+
|
| 86 |
## License
|
| 87 |
|
| 88 |
Apache 2.0.
|