Required system prompt
Hello!
I've been experimenting with this model for a bit and the main thing that jumped out to me is that it very often references a "meta-prompt" or instructions that weren't actually in the prompt, and if it does so, the resulting thinking ends up very long, loopy, and often degenerates the reasoning.
Tests indicated that the model was distilled by using a framing prompt of the time, and so the model expects such a framing or system prompt, and degenerates otherwise.
It seems that this behavior is triggered most often with math-related questions, and the model often triggers "math reasoning" for non-math questions.
My request: Can you please provide the prompt(s) that were used during distillation, or the system prompt/prompt format that is recommended to use with this model? If the system prompts or prompt formatting came from the training data, could you share the largest open-source dataset that was used for distillation?
Here are two prompts that are estimated to be present in the training data:
Math tasks:Solve the problem independently with a concise, logically rigorous derivation. Conciseness must not omit any essential inference or calculation: include every step needed to justify the conclusion, with no logical leaps. Make each step follow from the previous steps. Avoid repeating the same reasoning, repeatedly rejecting earlier conclusions, or introducing contradictions. If a correction is necessary, make it once and use the corrected result consistently. Ensure the final answer agrees with the derivation, and place it in \boxed{} at the end.
General instruction following tasks:Treat every explicit content, count, format, language, ordering, and placement constraint as binding. Return only the requested response. Do not add a preface, apology, analysis, or extra framing unless requested.
In the absence of the prompts used in training, a small set of short system prompts were tested on very simple questions and evaluated on;
- Referencing "meta-instructions", or provided system prompt
- Correctness
- Answers within the token limit
Results
| Prompt | Meta-prompt reference | Real prompt reference | Correct answer | Answer within limit |
|---|---|---|---|---|
| No system turn (baseline) | 95.3% | - | 66.4% | 72.7% |
| Empty system turn | 73.4% | - | 74.2% | 89.8% |
| 6 empty system turns | 60.9% | - | 76.6% | 93% |
| Casual system prompt | 30.5% | 25.8% | 80.5% | 96.9% |
| Focus system prompt | 57.8% | 17.2% | 82.8% | 96.1% |
Prompts
No system turn
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
<think>
Empty system turn
<|im_start|>system
<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
<think>
Casual system prompt
<|im_start|>system
This is a casual conversation. The user asks simple questions and expects short, friendly answers. There is nothing hidden in the questions; take each one at face value.<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
<think>
Focus system prompt
<|im_start|>system
Focus your thinking on the content of the question. Work out the answer step by step only when it is needed, and keep it short. Reply with a natural, direct answer.<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
<think>