"Use this model" boilerplate for Transformers doesn't work
I find your work very interesting and would like to explore what can be done with these very small models. I am trying to configure this model to run locally with transformers and have encountered some hiccups in the configuration. The chat template is not applied by the AutoTokenizer causing an error, so I manually set it to reflect the chat_template.json file.
At this point the eos_token and bos_token are still not set, and while checking the rest of the config I found conflicting information:
chat_template.json indicates the tokens should be <|im_start|> and <|im_end|>;config.json indicates "bos_token_id": 1, "eos_token_id": 2;tokenizer.json and tokenizer_config.json both indicate the token ids for 1 and 2 as <|begin_of_text|> and <|end_of_text|>; not <|im_start|> or <|im_end|>
However, the model card would indicate <|im_start|>/<|im_end|> as the intended tokens. I have limited experience with manual configuration, so I am unsure how to proceed; should I modify the tokenizer config, or the chat template? Manually setting tokenizer.bos_token="<|im_start|>"; tokenizer.eos_token="<|im_end|>" does allow the model to generate, however the output is severely degraded and runs slower than even my local Llama 8B model. This could also (likely) be due to another part of the configuration that wasn't applied by AutoTokenizer/AutoModel.
It would be great if the model could be updated to configure properly through AutoTokenizer/AutoModel, but if that's not currently a priority, I would appreciate more information on how it is intended to be configured in transformers (or another library). I suspect I am missing something fairly simple, so thank you for your time and patience with this issue!