| library_name: transformers | |
| tags: [] | |
| # Model Card for Model ID | |
| This is a tokenizer used in FAM model | |
| ## Tokenizer Details | |
| + **Architecture:** BPE | |
| + **Normalizer:** BertNormalizer | |
| + **Pre-tokenizer:** BertPreTokenizer | |
| + **Special tokens:** [UNK], [PAD], [CLS], [SEP], [MASK] | |
| + **Dataset:** | |
| + **Languages:** English | |
| Tokenizer training noteboook: | |