File size: 183 Bytes
1be11eb
 
 
 
 
 
 
1
2
3
4
5
6
7
{
  "tokenizer_type": "tiktoken",
  "encoding": "gpt2",
  "vocab_size": 50257,
  "install": "pip install tiktoken",
  "usage": "import tiktoken; enc = tiktoken.get_encoding('gpt2')"
}