miguelcsx
/

tolm-wordlevel-16k-tokenizer

Model card Files Files and versions

TOLM Word-Level Tokenizer (16k)

Whitespace-split word-level tokenizer with vocabulary size 16000, trained on the BabyLM Strict-Small cleaned corpus.

Special tokens: <unk>, <pad>, <s>, </s>, <mask>

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including miguelcsx/tolm-wordlevel-16k-tokenizer

WordLevel structured artifacts

Word-level tokenizer, encoded corpora, and structured-prior artifacts. • 4 items • Updated 13 days ago