yenupam/matra
Updated • 1
Matra
Indic specific most efficient tokenizor.
Note tokenizors with different vocab size and language combinations.
Matra Tokenizer Visualizer
🚀1Matra indic tokenizor playground.
Note interactive playground to test tokenizor