wfzyx

/

von

Like 63

The GitHub repository is archived as concluded, so this is for people still using
these weights rather than a bug report.

Measuring open System One models on public labelled sets, von's accuracy held up
well on four-way news classification, but the confidence it reports sits far below
how often it is right. A gate built on that number escalates most of what von
already answered correctly.

Setup: von-sdk 1.3.7 (reports von-1.3.0 weights), torch 2.14.1, Python 3.12,
Apple M4 32 GB, CPU, through von serve --device cpu. 150 rows of the ag_news test
split, a four-way choice with the four section names as options, sampled with a
fixed seed. Harness, per-row output and method:
https://github.com/gopaljigaur/system-one-bench

Accuracy 0.913, mean reported confidence 0.578. Every bin is low:

The 0.70 to 0.80 bin answered all 23 correctly while reporting 0.739. Expected
calibration error over the set is 0.336, all of it in the same direction.

What that costs a caller gating on it:

A gate at 0.8 sends 137 of 150 questions to a second, more expensive tier, and 126
of those already had the right answer.

The noul path looks fine. The same harness on 150 rows of sst2, a yes/no
noul, gives accuracy 0.847 at 0.024 calibration error, which is the best of the
six systems I measured. So this reads as specific to the choice path, not to the
model in general.

Two things I checked before reporting, so they are not the explanation:

• The calibration file is loading. OptionMarkerBackend._get_model() resolves it
  from the Hub alongside the weights, and the instance comes up with
  _default_temp = 2.2 and the input-conditioned map active
  (bias 2.0151, entropy -3.2369, log_tokens 11.4916, n_options -3.5574,
  bounds 0.3 to 12.0). This is not #13 coming back.
• I scored the confidence field von returns, not the top probability. They are
  different: on one row von answers Business with probabilities
  {World 0.0302, Sports 0.0295, Business 0.9077, Sci/Tech 0.0325} and
  confidence 0.877. Scoring the top probability instead gives 0.230 calibration
  error, still in the same direction.

The n_options coefficient is where I would look first, since four options is the
only structural difference from the two-option noul case that calibrates well,
but I have not tried refitting and I do not know the fitting data, so treat that as a
guess, not a finding.

Nothing here asks anyone to change anything. It is worth knowing before you pick a
threshold off these weights.

Disclosure: the harness drives every system through pydecide, which is my own
client, used here only as plumbing; swapping it out would not change which answer
von returns. These runs and this write-up were done with an AI assistant.