Sometimes not thinking enough

#2
by idleDaemon - opened

I understand that this model has a method of choosing when to reason or not - but it appears to get stuck sometimes and never reason... So I guess I'm encountering the opposite of what others are! (Yes, thinking is enabled, and I can see it happening in many interactions.)

Some of my chats will be great, with plenty of reasoning, then others will have a single thinking block then it'll never reason again. It will perform some short-hand reasoning in the plain output, but no long thinking blocks. This isn't just leaked reasoning either - no evidence of the elements/tokens in the normal output.

Does the determination of whether it should think or not stop happening after the first message, or are the odds somehow heavily biased by it? I seem unable to trigger it back into reasoning once it's stopped.

On the positive side, I've yet to get it looping, even up to 500K context length chats with lots of large XML and C++ manipulation - so yay for the fast fixes :)

EDIT: It will still loop sometimes in reasoning blocks :(
I've yet to get it looping outside of one though.

Also, after further trials, I think my hunch is correct: If it starts out thinking, it continues to do so in later turns, but if it didn't start with thinking after the first message, I cannot trigger it.

For example, if I start a chat like this:

User: Respond with the letter A
Agent: A
User: Now, identify the longest prime number that's contained in the first 1000 digits of pi

The agent continues to never use blocks.

But, if instead I start with "Think about the number pi, identify the first prime in it, then respond with the letter A and the first prime in pi.", that triggers a reasoning trace and it will continue reasoning throughout the chat.

I've noticed that this isn't consistent though - so whatever determines if it should think or not on the first response isn't deterministic based on the prompt alone.

But once it has reasoned, it continues to do so for every turn. If it didn't reason on the first turn, I'm out of luck.

Sign up or log in to comment