tspersian
/

mana_tokenizer

Model card Files Files and versions

mana_tokenizer / test.py

tspersian's picture

package

d786ff1 about 1 year ago

history blame contribute delete

229 Bytes

	from .mana_tokenizer import ManaTokenizer
	tokenizer = ManaTokenizer()
	text = "سلام من یک متن تست برای تست این تست هستم."
	print(tokenizer.encode(text))
	print(tokenizer.decode(tokenizer.encode(text)))