harpertoken commited on
Commit
efe4e36
·
verified ·
1 Parent(s): 407939c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -11,7 +11,7 @@ tags:
11
 
12
  # word
13
 
14
- A GPT-2 language model trained from random initialisation on WikiText-2, rather than adapted from OpenAI's released weights. The architecture is GPT-2 small — twelve layers, 768 embedding dimensions, 50257-token vocabulary — which works out at about 124M parameters; the previous version of this card put the figure at 117M, which does not match the 496 MB float32 checkpoint. `config.json` carries no `_name_or_path`, confirming nothing was fine-tuned.
15
 
16
  The tokenizer is GPT-2's own byte-pair vocabulary, taken from [`openai-community/gpt2`](https://huggingface.co/openai-community/gpt2), because the checkpoint was trained from scratch and therefore ships no vocabulary of its own. It matches: both are 50257 entries, so the token ids in the weights line up with this tokenizer exactly. An earlier version of this repository shipped no tokenizer at all, and `AutoTokenizer` then returned an empty vocabulary of length one rather than raising, which meant every input collapsed to the unknown token and generation returned noise without any error. That is fixed. GPT-2 uses byte-pair encoding without a distinct unknown token, so rare inputs are segmented rather than discarded — one reason a model of this size produces fluent-looking nonsense.
17
 
 
11
 
12
  # word
13
 
14
+ A GPT-2 language model trained from random initialisation on WikiText-2, rather than adapted from OpenAI's released weights. The architecture is GPT-2 small — twelve layers, 768 embedding dimensions, 50257-token vocabulary — which works out at about 124M parameters; the previous version of this card put the figure at 117M, which does not match the 496 MB float32 checkpoint. `config.json` carries no `_name_or_path`, confirming nothing was fine-tuned. One structural difference from stock GPT-2 is worth knowing: `n_positions` is 512 rather than 1024, and the position embedding really is 512 rows, so this model accepts half the context a standard GPT-2 does.
15
 
16
  The tokenizer is GPT-2's own byte-pair vocabulary, taken from [`openai-community/gpt2`](https://huggingface.co/openai-community/gpt2), because the checkpoint was trained from scratch and therefore ships no vocabulary of its own. It matches: both are 50257 entries, so the token ids in the weights line up with this tokenizer exactly. An earlier version of this repository shipped no tokenizer at all, and `AutoTokenizer` then returned an empty vocabulary of length one rather than raising, which meant every input collapsed to the unknown token and generation returned noise without any error. That is fixed. GPT-2 uses byte-pair encoding without a distinct unknown token, so rare inputs are segmented rather than discarded — one reason a model of this size produces fluent-looking nonsense.
17