CA3 All use casesContact
AI judges · continual learning · Qwen3.5-9B and Strands Decider 2B

AI judges that learn: a person reviews only 7 checks in 100.

Every new fact a model learns after deployment is checked before it is saved. AI judges settle 93 checks in 100 with no person. And they learn: a small judge learns from the big one to settle two thirds of the checks, at a fraction of its cost; the doubtful ones still go to a person.

Results recorded from 21 to 29 October 2026
Of every 100 checks in the final test: 66 settled by the small judge, which learns from the big one; 27 by the big judge; 7 by a person, the doubtful ones
To a person7 in 100229 of 3,136 checks, on three models
Small judge66%of the checks, settled before the big judge reads them
Planted falsehoods0 of 42that the big judge stopped were let through by the small one
Careless reviewerWeek 3flagged; its mistakes stay out of what the small judge learns
Holds

Two thirds settled by the small judge, and none of the planted ones.

1,298 checks on news and models the small judge had never seen: later news (November and December 2020) on Qwen2.5 7B, and two other models, Qwen3 14B and Nemotron 3 Nano 30B. Each dot is one check; ringed in red, the planted falsehoods the big judge stopped: all went to a person.

Each dot is one check: 300 of 463, 261 of 392 and 295 of 443 settled by the small judge; the 12, 12 and 18 planted falsehoods the big judge stopped all went to a person
65 to 67%of the checks settled by the small judge, depending on the source
0 of 42planted falsehoods let through
0 of 20known-bad items settled, from two test banks
Learns

Week by week, it takes over.

Twenty-four simulated weeks of news. For eighteen weeks the small judge only collects the big judge's verdicts; then it starts settling checks, and with more verdicts it settles more. Part of what it settles is checked again: in every week it was in use, nothing was found, and no planted falsehood was let through.

Share of each week settled by the small judge: none in weeks 1 to 18; 39, 29, 30 and 25% in weeks 19 to 22, learning from 673 verdicts; 64 and 70% in weeks 23 and 24, from 801
25 to 39%of each week, learning from its first 673 verdicts
64 and 70%from 801 verdicts, the last two weeks
27 to 65%on the later news in the test, from its first version to its last
Supervises

It checks its reviewers.

The reviewer is the person who decides the doubtful cases. If that person makes mistakes, the small judge must not learn them. We simulated two careless reviewers: one who approves everything and one who errs 1 time in 5. The first was flagged in week 3, the second in week 4, the good one never; their mistakes stayed out of what the small judge learns. Flagging does not undo a reviewer's own decisions.

Week 3a reviewer who approves everything is flagged
Week 4one who errs 1 time in 5
0 of 42planted falsehoods let through by the small judge in either case, for every type of fact
Saves

The big judge is expensive; the small one is cheap.

In our tests the big judge took 27% of the learning compute on Qwen3 14B and 60% on Nemotron 3 Nano, counting its active parameters. Reading the same checks costs the small judge about 5% of what it costs the big one. A first version, settling about 55% of the checks, made learning 10% cheaper on the 14B and 26% on the Nano (6% counting all its parameters).

27 · 60%of the learning compute goes to the big judge, on the 14B and on the Nano
5%what the small judge costs to read the same checks
10 · 26%cheaper learning, settling about 55% of the checks
How the test was run

Five gates in three days, with open models.

The comparison is the big judge alone, reading every check (Qwen3.5-9B, thinking). Same checks, same planted falsehoods, same rule: a planted falsehood is let through if the small judge settles a check the big one stopped. Two test banks add 20 items known to be bad. The criteria were fixed in writing before each test.

27 OctOff-the-shelf judgePlanted falsehoods got through.discarded
27 OctTrain on verdictsOn decisions already made.settles about 55% · 0 let through
28 Oct24 weeks of newsLearning week by week.settles 65 to 67% · 0 let through
28 OctCareless reviewersTwo, simulated.flagged in week 3 or 4
29 OctSettle doubts aloneA second look did not hold.stays with a person
Small judgeLater newsQwen3 14BNemotron NanoReviewer flagged
Learns from the big judge64.8%66.6%66.6%–
Learns from the person23.3%17.9%16.5%never
Same, with a client rule as code18.6%13.3%12.9%never
Reviewer errs 1 time in 564.8%66.6%66.6%week 4
Reviewer approves everything64.8%66.6%66.6%week 3

Share of checks the small judge settles before the big one reads them, in the final test. In every row the small judge let through 0 of the 42 planted falsehoods and settled 0 of the 20 known-bad items. The reviewer is the person who decides the doubtful cases, here simulated. The client rule (no check with a figure settled by the small judge) held.

Learning from the big judge's verdicts, the small judge settles two thirds of the checks on news and models it had never seen, and lets through none of the 42 planted falsehoods the big judge stopped.
It improves with experience, and a careless reviewer's mistakes stay out of what it learns: one is flagged in week 3 or 4. Checks with figures are settled less often.
The person stays in control and is asked little: 7 checks in 100, the doubtful ones. That figure is set by the big judge; the small one saves its compute.

Test it on your data.

A replay on your own data, on our GPUs: CA3 checks every new fact, and we measure what would have entered, what was stopped, how many checks reached a person and what checking costs. CA3 runs on our side and stays confidential; the outcome is a joint evaluation report.

ContactAll use cases

Model and data

  • Models that learn: Qwen2.5 7B (five windows of news), Qwen3 14B and NVIDIA Nemotron 3 Nano 30B-A3B.
  • Big judge: Qwen3.5-9B, thinking. Small judge: Strands Decider 2B (AWS Strands Labs, open weights), trained on the big judge's verdicts.
  • News: StreamingQA on WMT News Crawl 2020 [1], with planted falsehoods; two test banks of 10 good and 10 bad items each.
  • Each test on one GPU at most (NVIDIA H200 or RTX 6000 Ada).

Sources

  • [1] Liška A, Kočiský T, Gribovskaya E, et al. StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models. ICML 2022.
  • 7 in 100: 229 of 3,136 checks reached a person (167 of 2,301 on Qwen2.5 7B, 28 of 392 on Qwen3 14B, 34 of 443 on Nemotron Nano). Final test: of 1,298 checks, 856 settled by the small judge, 345 by the big judge (20 of them repeats of a case a person had already decided) and 97 by a person.
  • Tests run from 27 to 29 October 2026, with the criteria fixed in writing before each one; the data, from 21 October.

Limits

  • A simulated client: real news with planted falsehoods and three models that learn; the person and the careless reviewers are simulated.
  • Few planted falsehoods per block (12 to 18 per model in the test): zero let through is not yet a rate.
  • The bar is the big judge: we measure what the small one adds, not the big one's own misses. In the final test the small judge settled 6 checks the big one would have stopped, none of them planted.
  • Flagging a reviewer keeps their mistakes out of what the small judge learns; it does not undo their own decisions.
  • What did not hold: an off-the-shelf small judge let planted falsehoods through; learning from the person's decisions settles less (17 to 23 in 100), because the person answers another question; and a second look that would settle doubtful cases alone did not hold (one version let 1 in 20 through, another settled none): a credible false update reads like a true one. The doubtful ones stay with a person; the small judge saves the big judge's work, not the person's.

Results from recorded tests. The mechanism is not shown. Model and organization names are for identification only. Independent work by Biome Makers, not produced or endorsed by AWS, Alibaba, NVIDIA or the authors cited. CA3 is developed by Alberto Acedo, PhD, at Biome Makers Inc.