Model Card for Model ID
This is a tokenizer used in FAM model
Tokenizer Details
- Architecture: BPE
- Normalizer: BertNormalizer
- Pre-tokenizer: BertPreTokenizer
- Special tokens: [UNK], [PAD], [CLS], [SEP], [MASK]
- Dataset:
- Languages: English
Tokenizer training noteboook:
