Models
Datasets
Spaces
Buckets new
Docs
Enterprise
Pricing
- Website
- Community
- Solutions
Log In
Sign Up

Norod78
/

gpt-j-hebrew-tokenizer

Model card Files Files and versions

gpt-j-hebrew-tokenizer

5.97 MB

Ctrl+K

Ctrl+K

1 contributor

History: 4 commits

Norod78's picture

Added tiktoken extension for this tokenizer

0f53788 about 3 years ago

.gitattributes

1.48 kB
initial commit about 3 years ago
merges.txt

1.11 MB
Updated eos_token to have the id of 50256 about 3 years ago
special_tokens_map.json

438 Bytes
Upload tokenizer about 3 years ago
tiktoken_ext_norod78_hf.py

753 Bytes
Added tiktoken extension for this tokenizer about 3 years ago
tokenizer.json

3.41 MB
Updated eos_token to have the id of 50256 about 3 years ago
tokenizer_config.json

766 Bytes
Updated eos_token to have the id of 50256 about 3 years ago
vocab.json

1.45 MB
Updated eos_token to have the id of 50256 about 3 years ago