| | --- |
| | license: mit |
| | language: |
| | - en |
| | tags: |
| | - babylm |
| | - tokenizer |
| | datasets: |
| | - nilq/babylm-100M |
| | --- |
| | |
| | ## Baby Tokenizer |
| |
|
| | Compact sentencepiece tokenizer for sample-efficient English language modeling, simply tokenizing natural language. |
| |
|
| | ### Usage |
| |
|
| | #### Transformers |
| |
|
| | ```py |
| | from transformers import AutoTokenizer |
| | |
| | tokenizer_baby = AutoTokenizer.from_pretrained("nilq/baby-tokenizer") |
| | ``` |
| |
|
| | #### Tokenizers |
| |
|
| | ```py |
| | from tokenizers import Tokenizer |
| | |
| | tokenizer_baby = Tokenizer.from_pretrained("nilq/baby-tokenizer") |
| | ``` |
| |
|
| | ### Data |
| |
|
| | This tokeniser is derived from the BabyLM 100M dataset of mixed domain data, consisting of the following sources: |
| | - CHILDES (child-directed speech) |
| | - Subtitles (speech) |
| | - BNC (speech) |
| | - TED talks (speech) |
| | - children's books (simple written language). |
| |
|
| | ### Specifications |
| |
|
| | - Vocabulary size: 20k |
| | - Alphabet limit: 150 |
| | - Minimum token frequency: 100 |
