Spaces:
Running
Instruct models receive raw prompt without chat template
Hi! I noticed that instruct models in SLM Arena receive the raw prompt without their chat template applied. For example, my model ThingAI/Quark-135m uses <|user|>...<|end|><|assistant|> and generates gibberish without it.
A simple fix in _prepare_inputs would be to check if the tokenizer has a chat_template and apply it for instruct models:
if is_instruct and hasattr(tokenizer, 'chat_template'):
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
This would fix it for all instruct models in the catalog, not just mine. Happy to open a PR if you prefer.
Thanks for flagging this! I’ve implemented the fix.
Instruct models in the catalog now apply their tokenizer’s chat template with a user message and generation prompt, while base models retain the existing raw-prompt behavior. If a template is missing, broken, or unsupported, the code safely falls back to the raw prompt.
This should resolve the issue for ThingAI/Quark-135m and other instruct models with the same requirement.