Changing reasoning effort requires rebuilding whole kv cache
Hi! This really is an amazing model so congrats first and foremost!
If there's a small gripe I have with it tho is that changing the thinking effort (for example from xhigh to low) implies changing a very small line at the top of the context window, so if you have already a 100k context built, you have to rebuild everything from scratch.
Would it be possible to train the model to "see" these small settings, such as reasoning effort, in something akin to a system prompt put right after the conversation history and right before the LLM output?
I know you don't allow system prompts that aren't at the very top of the context window for security reasons, but how about then a "<|im_start|>settings\n" tag to be put right before the output?
Rebuilding hundreds of thousands of tokens of kv cache isn't very practical for local inference, and it probably isn't great for servers either to be honest.
Cheers!
Cloud models like Claude Opus do the same thing, changing effort reprocesses the whole conversation history. I doubt there's a way around it.
If you don't want to include it at the beginning of the context window by enabling a reasoning mode, you can instead tell the model how to reason on an ad-hoc basis, such as the general purpose "Max" reasoning prompt DeepSeek V4 uses: https://arxiv.org/html/2606.19348v1#S5.T3 or Mistral's "# HOW YOU SHOULD THINK AND ANSWER\n\nFirst draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input.\n\n"Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response to the user."" https://docs.mistral.ai/resources/deprecated/native-reasoning.
You can store these sorts of things as prompts snippets.