cbrownpinilla
/

bottom-3B

Model card Files Files and versions

bottom-3B / tokenization_olmo_fast.py

cbrownpinilla's picture

HF integration

5fc231f 10 months ago

history blame contribute delete

187 Bytes

	from hf_olmo.tokenization_olmo_fast import OLMoTokenizerFast, OLMoConfig
	from transformers import AutoTokenizer

	AutoTokenizer.register(OLMoConfig, fast_tokenizer_class=OLMoTokenizerFast)