Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
jbrashear 
posted an update 30 days ago
Post
125
I've been training Jebadiah, small models that don't chat. You give one a typed question about some JSON state and it gives you back a probability for every option, from one forward pass.

There are three sizes, 4B, 9B and 27B, all Apache 2.0, with GGUF and MLX builds. Each repo ships a temperatures.json fitted on held-out data, so the probabilities are calibrated per question type.

New this week: pip install jebadiah-decide, then jeb serve. It runs them through Ollama, LM Studio, llama-server, vLLM or MLX and puts /v1/systemone and /v1/decide on localhost. The setup for each runtime is here: https://github.com/getainode/jebadiah/blob/main/docs/run-locally.md

If you try it on a routing or triage step of your own, I'd like to hear how it does.

https://huggingface.co/collections/frontier-infra/jebadiah-open-system-one-decision-models-6ab80765ddd3fa0b3eba5213
dipankarsarkar

Your 27B score refit carries over to the 4B and 9B. Except on HelpSteer2, where it doubles ECE.

The 4B and 9B v2 still ship the train score temperature (1.20, 1.22), the fit the 27B card replaced on 09-26. I re-tempered their own eval records with each repo's hard score fit (0.83 for both), choice and noul untouched, 10 bins, repeat 0. As a check, the same code gives the 27B table's typed-decisions and Nimble 324 rows to 0.001, and the 4B/9B stored ECE on these sets to 0.003.

ECE, train -> hard score T    4B              9B              27B (your card)
typed-decisions              0.171 -> 0.135  0.156 -> 0.121  0.175 -> 0.134
Nimble 324                   0.090 -> 0.069  0.085 -> 0.070  0.145 -> 0.124
Kev decision-v7              0.044 -> 0.055  0.047 -> 0.048  0.027 -> 0.040
Jevals HelpSteer2            0.049 -> 0.073  0.038 -> 0.077  0.078 -> 0.062

The first three rows are the 27B story again. The last one flips. On the 27B, sharpening undid what the train fit had cost HelpSteer2. On the 4B and 9B, train was already their best fit there (better than no temperature), and sharpening costs it.

So "score wants sharpening" holds on the mixed sets at all three sizes, but not on HelpSteer2 for the small ones.

One caveat. On the Nimble public subsets I can't rebuild the stored overall from the rows (9B HelpSteer2: 0.060 from the rows against 0.073 stored, same accuracy, different bin counts), so I left SummEval out.

If you refit the small models, would you take the 27B's mixed rule and eat the HelpSteer2 loss, or split score by rubric on the calibration split?

·
jbrashear

Dipankar, thank you for this. You went back to the per-question records and re-tempered them yourself, which is exactly why we put them up, and you caught something real. The 4B and 9B are still shipping the score temperature fit to the training target. The 27B isn't.

Your numbers check out. We rebuilt every one of them from the eval records in the repos.

The mismatch you hit on the Nimble public subsets is binning. Our ECE rounds the confidence to a whole percent before it picks a bin, so round(100c) // 10, and your rows look like floor(10c). On these score sets a lot of the confidence sits right on the bin edges, so it matters more than you'd think. With your bins the 9B Nimble HelpSteer2 comes out 0.060. With ours it's 0.073, and the rows rebuild the stored number exactly. That same difference explains the small gaps in your 4B and 9B columns.

Your 27B row came from our card, so it's in our binning. In yours the 27B Kev decision-v7 goes 0.026 to 0.039 and Jevals HelpSteer2 goes 0.078 to 0.057. None of that changes your conclusion.

On your question, we're taking the 27B rule for the 4B and 9B and eating the HelpSteer2 loss.

I did want to know if splitting score by rubric would buy it back. So we pulled raw logits for all 549 score questions in the calibration split, fit on one half, scored on the other half, and did that 200 times.

It doesn't buy it back. The HelpSteer2 part of the calibration split wants about 0.84 on its own, the same as the pooled fit.

What's going on is the labels. The HelpSteer2 questions in our calibration split are spread pretty evenly across 0 to 4. The Jevals and Nimble HelpSteer2 items are over 70% 3s and 4s. The model is less accurate on that mix and it's too sure of itself there, so it wants to be softer. That's a shift in the label mix, and no rubric split of our calibration data can see it.

Splitting by rubric did help our own typed decision rubrics, which want something closer to 0.5. But the SummEval part of the split is only four articles, the finer per rubric fits swung anywhere from 0.8 to 15 on groups of 17 questions, and a request that hits the server doesn't carry a rubric key anyway. So personally I don't want to ship that off a split this small.

Here's what we're shipping. The 4B and 9B get the hard score fit, 0.83, with choice and noul left alone, same as the 27B. Accuracy doesn't move. The cards will say plainly that HelpSteer2 on both sizes and Kev decision-v7 on the 4B get worse, and why, and that if your traffic looks like HelpSteer2 you should refit on your own labels. Per rubric calibration goes on the list for v3, with a bigger calibration split.

Thanks again. This is exactly the kind of review I was hoping for when we published the records. :)
dipankarsarkar

Your binning checks out. With round(100c) // 10 the rows rebuild every stored ECE I tried, 21 of 21 across the three sizes, calibrated and raw. Your two floor-binned 27B cells come out 0.026 -> 0.039 and 0.078 -> 0.057 here too.

That also lets me put SummEval back. Sharpening helps it at every size:

ECE, train -> hard score T (your bins)   4B              9B              27B
SummEval consistency                     0.184 -> 0.112  0.195 -> 0.118  0.165 -> 0.109
SummEval relevance                       0.054 -> 0.035  0.084 -> 0.033  0.072 -> 0.050

One correction to my own table. The 27B loses on the Nimble public HelpSteer2 subset too, 0.049 -> 0.061. I only had Jevals for it.

Then I tried to see the label shift directly. Same records, repeat 0, but each HelpSteer2 set reweighted to a flat 0-4 mix (weight 0.2 / p(label)). If the 71-72% share of 3s and 4s were the whole cause, the loss should go away. It shrinks, about by half on the small models, but it never flips:

HelpSteer2 ECE, train -> hard T    as shipped        flat 0-4 mix
4B  Jevals (300)                   0.048 -> 0.074    0.035 -> 0.046
4B  Nimble public (249)            0.049 -> 0.097    0.092 -> 0.120
9B  Jevals                         0.039 -> 0.082    0.047 -> 0.064
9B  Nimble public                  0.073 -> 0.098    0.101 -> 0.114
27B Jevals                         0.078 -> 0.062    0.075 -> 0.060
27B Nimble public                  0.049 -> 0.061    0.052 -> 0.093

Accuracy barely moves at the flat mix (0.38 to 0.43 on the four small-model cells, against 0.39 to 0.43 as shipped). As shipped, the train T already puts mean confidence within 0.02 of accuracy on all four small-model cells. The hard T lifts it by 0.06 to 0.07.

Caveat: flat weighting puts 2.6 to 2.9x on the 17 to 23 label-0 rows, so it is noisy.

So the mix is part of it, not all of it. The rest looks like the items. Your card says the calibration split is the 5% side of a 95/5 split over the training data, so its HelpSteer2 items come from the same train split the model learned from.

What accuracy does the HelpSteer2 part of the calibration split get? If it is well above the 0.43 these sets top out at at a flat mix, a held-out HelpSteer2 validation slice might be the cheap test before v3.

·
jbrashear
•
edited 29 days ago

Thanks again, Dipankar. Good catch on your own 27B row too. We get 0.049 to 0.061 on Nimble public HelpSteer2 as well.

I went back through your SummEval numbers and the mean confidence point against our records. Consistency and relevance, all three sizes, match to the third decimal. So does the confidence gap: at the train T the small models sit within 0.02 of their accuracy on both HelpSteer2 sets, and the hard T pushes that up by 0.06 to 0.07. Your flat mix result matches ours too. The loss roughly halves and never flips.

On your question. The HelpSteer2 part of the calibration split is 161 questions over 87 prompts, and on all of it the 4B gets 0.50 and the 9B gets 0.48.

That number is misleading though. 68 of those 161 are the other four HelpSteer2 attributes (coherence, complexity, verbosity, correctness), which Jevals and Nimble never ask, and coherence and complexity run around 0.7. On helpfulness alone, the question your two eval sets actually ask, it's 0.42 on the 4B and 0.40 on the 9B. Reweighted to a flat mix it's 0.44 and 0.41.

So it isn't well above your 0.43. It's about the same.

You're right about where it comes from. The calibration split is the 5% side of a 95/5 split over the training pool. The HelpSteer2 items in it come from HelpSteer2's train split, drawn evenly across the helpfulness levels. The split is by prompt, so both responses to a prompt land on the same side and no calibration prompt was ever trained on. Jevals and Nimble both sample HelpSteer2's validation split, and the pool builder throws out every validation row.

Here's the part I didn't expect. At the same accuracy, the model is less sure of itself on those calibration items than on the eval items. Fit on helpfulness alone, the calibration slice wants T 0.79 on the 4B and 0.83 on the 9B. A family bootstrap puts the 90% range at 0.64 to 0.99 and 0.67 to 1.04. Fit on Jevals and Nimble directly, the same models want 1.05 to 1.20.

So the calibration fit really is off for HelpSteer2 traffic. It just isn't off because the split is in distribution and flatters the accuracy. Something about that draw makes the model less confident than it is on the validation data, and the label mix is only part of it, same as you found.

The cheap test is sitting right there. HelpSteer2's validation split has 418 rows, over 209 prompts, that neither Jevals nor Nimble uses. They're at the natural label mix and never went near the training pool. We pull raw scores for those from the fleet, fit the score T on them, and see which side it lands on. If it comes out near 1.1, the calibration draw is the problem. If it comes out near 0.8, the two eval subsets are the odd ones. It's a few hundred requests per model, so we'll run it before we touch v3.

What goes on the v3 list: the calibration split draws HelpSteer2 at the natural label mix instead of evenly across levels, a held-out validation slice stays in the eval suite as a standing calibration check, and the calibration split gets bigger than 87 HelpSteer2 prompts.

I really appreciate you pushing on this. :)
In this post