the-tokenizer-playground

Running

the-tokenizer-playground / README.md

Xenova HF Staff

Update README.md

618db5f verified almost 2 years ago

284 Bytes

	---
	title: The Tokenizer Playground
	emoji: 📝
	colorFrom: indigo
	colorTo: purple
	sdk: static
	pinned: false
	short_description: Experiment with and compare different tokenizers
	---

	Link to source code: https://github.com/xenova/transformers.js/tree/main/examples/tokenizer-playground.