Tokenizer training data
#1
by qikp - opened
Where is the tokenizer training data?
The tokenizer is a custom BPE tokenizer (vocab size 1024) trained from scratch on a subset of Fineweb-Edu, which is the same dataset the model was trained on. The trained tokenizer is included in the repo as tokenizer.json.
Answered! The tokenizer is a custom BPE tokenizer (vocab 1024) trained from scratch on Fineweb-Edu data.
QyrouNnet-AI changed discussion status to closed