Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
mike-ravkine 
posted an update Apr 17
Post
118
A word of warning that my initial run of 3.6 produced unfavorable results vs 3.5; performance on out-of-distribution and instruction following tasks appears to have collapsed. Potentially a vLLM 0.19 issue here, the original eval was done with their fork of 0.18. I am re-running both with nightly so we have apples to apples and will report back, curious to get to the bottom of this one.
mike-ravkine
•
edited Apr 18

Some clarity is emerging:

image

The distribution of response lengths has shifted considerably in 3.6 and 2 of my tasks are no longer fitting into 16k, the ignorance zone blows up.

Re-running at 32k then we'll see if that extra thinking pays off or nah.

An interesting outlier here is the word-sort task where 3.6 thinks ~half as much and this costs it about 10pp of performance.
In this post