metadata
datasets:
- phonemetransformers/IPA-BabyLM
language:
- en
From Babble to Words: Tokenizers
Tokenizers trained for From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes.
This repository contains the eight tokenizers trained for the project, covering the combinations of the three transformations:
- Character-based tokenization (
CHAR) vs. subword tokenization (BPE) - Tokenizer for phonemic data (
PHON) vs. orthographic data (TXT) - Tokenizer removes whitespace (
SPACELESS) vs. keeps whitespace
To load a tokenizer:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('phonemetransformers/babble-tokenizers', subfolder='BABYLM-TOKENIZER-CHAR-TXT')