Add <|im_assistant|> to eos_token_id: [163584, 163586] e46ff0f verified prince-canuma commited on Jan 27
Set eos_token to <|im_end|> for proper chat termination eb07330 verified prince-canuma commited on Jan 27
Fix media_placeholder_token_id to match converted tokenizer 65c3725 verified prince-canuma commited on Jan 27
Add pre-converted tokenizer_config.json (no tiktoken dependency) d07b2ad verified prince-canuma commited on Jan 27
Add pre-converted tokenizer.json (no tiktoken dependency) 891bf91 verified prince-canuma commited on Jan 27