Commit History
fix(tokenizer): Sets fim and think tokens to non-special, and removes unk_token. 02fcf71 verified
fix(tokenizer_config): Sets fim and think tokens to non-special, and removes unk_token. 6265f82 verified
fix(special_tokens_map): Removes unk_token. 620a83c verified
chore(root): Adds top_k information even if 50 is already the default. 57faa53
Gustavo de Rosa commited on