Introducing Bosun

Stephen Solka

stephen-solka

Follow

Hanno-Labs

Links: Bosun-XS (0.6B) · Bosun-4B · WarrantBench

TL;DR

The judge that keeps an agent's memory — its knowledge graph — clean. As an agent accumulates memory as a graph of facts linked by relationships, Bosun-XS decides, edge by edge, which connections are warranted — supported, non-redundant, still-true — so the graph stays useful instead of growing into noise that drowns the model reading it back. Nothing else scores that "judge" step; Bosun-XS is a small, fast, calibrated model built for it, and you program it with a sentence.

Given two findings and an instruction, it emits P = sigmoid(logit_yes − logit_no) ∈ [0, 1] — how strongly the pair satisfies the rule you supplied, with no opinion of its own. "Warranted" isn't one fixed rule (same-entity, cross-domain bridge, not-a-duplicate, still-supported-by-evidence), so you define it per graph; Bosun-XS follows the rule, respects negation, and generalizes to rules it never trained on. That same capability is exactly what RAG filtering, content moderation, and deduplication need too — knowledge-graph curation is simply where the need bites first and hardest.

Bosun-4B is the same idea with a bigger brain. On FollowIR, the public benchmark for instruction-following retrieval, Bosun-4B ranks first outright — beating the previous best (a 7B trained on the benchmark) by five points — and Bosun-XS, at 0.6B, ranks third, ahead of Mistral-7B-instruct. We're open-sourcing both (weights →) and WarrantBench, the eval that measures them (eval →).

One question, everywhere downstream

Curating a graph is one instance of a question every system asks, millions of times a day: does this thing satisfy this rule? Keep this memory or forget it. Are these two findings the same fact. Does this source still support that claim. And past the graph: keep this RAG chunk or drop it, does this post break this policy, are these the same company.

There are two ways to answer it today, and both are broken. Pay a frontier LLM per item — and accept that you cannot run it on the volume you have, that it drifts run to run, and that you have outsourced your core judgment to someone else's endpoint. Or use something cheap that cannot read the rule — cosine, a static classifier, a similarity score — and accept that it is guessing. The industry has been choosing between an answer it can't afford and an answer that isn't one.

Bosun-XS is the third option, and it is the one that should have existed all along: a judge you reprogram with a sentence, that returns a number you can threshold, runs deterministically, and costs a forward pass.

What it is

Bosun is opinion-free. It holds no view of its own about what a good answer is — no baked-in notion of relevance, no fixed idea of what connects two things. You bring the opinion, as a sentence, and Bosun renders the verdict. The <Instruct> slot is the opinion; the model is the engine that executes it.

Mechanically: a LoRA fine-tune of Qwen3-Reranker (0.6B for Bosun-XS, 4B for Bosun-4B), a cross-encoder. The input is one string — <Instruct> (the rule), <Query> (the claim), <Document> (the evidence) — and one forward pass reads the model's native logits at the final token: score = sigmoid(logit("yes") − logit("no")). One calibrated number, no text. We train it by soft-target distillation, so the output is trust, not an argmax.

And here is the part that matters: we trained a ranker, but Bosun doesn't rank. Ranking is its heritage and its credential — it's a reranker, and reranking is where the public yardsticks live, so that's where we prove it. But the job isn't ordering a list; it's one verdict, on one pair, against the opinion you supplied. That's why you run it on every edge instead of a batch — and why the economics work at all.

WarrantBench: following relevance vs executing an opinion

Instruction-following retrieval already has benchmarks — FollowIR chief among them, the one Bosun tops. But FollowIR measures one specific thing: whether you follow a relevance instruction, a description of what makes a document relevant to a query. Relevance is a fine opinion. It is also just one opinion — the IR community's — baked into the benchmark.

WarrantBench measures the other thing: whether a judge executes an arbitrary opinion, including ones that have nothing to do with relevance. "Connected only if they share a specific entity." "Connected only if they're about different topics." "Connected only if both mention a figure over $1B." These aren't relevance narratives; they're rules — and a moderation policy, a dedup definition, and a memory-retention criterion are rules too. WarrantBench is the benchmark for a judge that has no opinion of its own.

Its headline metric makes that concrete: one fixed pair, two opposite rules, and the score must flip the right way. A cosine baseline scores zero — it can't read the rule, so it can't move. The gold is construction-truth, frozen, scored with no model in the loop, so it reruns free and identical forever — and it deliberately tests rules the model never trained on, to prove the following generalizes, not the rules. We're releasing it as the companion to FollowIR: relevance is covered; this covers everything else you might think to ask.

Results

Bosun out-steers a frontier LLM. On WarrantBench:

It reads the rule and obeys it. A larger model carries a stronger but more diffuse prior; on the axis that matters — following the instruction you gave — the small specialist wins.

It follows rules it has never seen. Bosun trained on four relation families. Given three it never saw, it applies them cold:

Training it to follow some rules made it follow new ones. That is a general judge, not a task list.

FollowIR. FollowIR measures whether a retriever changes its judgments correctly when the instruction changes — the public version of our steerability axis. Most retrievers score zero or negative: they treat the instruction as keywords. Both Bosuns top the board:

Bosun-4B is state of the art — five points clear of the previous best, a 7B model trained on the same benchmark, and first on all three tasks. Bosun-XS, at 0.6B, sits third — ahead of a 7B instruction-tuned LLM. Both are ours, and the lift is the training, not the base: the untuned base scores +4.5.

Two results live in that number, and both are true. Bosun trained on FollowIR's training split — the same data the 7B leader used — to reach +10.5. The previous version, never shown FollowIR at all, scored +7.6 zero-shot — already the best reranker on the board. It learns the benchmark when you let it; it nearly tops the benchmark when you don't.

The frontier model is a line item you cannot pay

A frontier LLM, scored one document at a time, judges relevance better than Bosun does. This is not the weakness it sounds like — it is the whole argument.

To measure that per-document margin at all, we had to cap the LLM to 100 candidates per query. The real pool is a thousand. Pointwise LLM reranking is one API call per document; the full pool is a hundred thousand calls per task — a bill, a rate limit, an afternoon of latency. Bosun reranked the full pool — a hundred thousand forward passes — on a single GPU, in minutes, deterministically, for cents.

The frontier model wins on the pool small enough to afford it. Bosun wins on the pool you actually have. Per-document quality is the wrong axis the moment you cannot run the model on the documents.

And it matches the LLM head-to-head

The leaderboard is one comparison. The harder one is the frontier LLM itself, scored one document at a time — and that's the comparison we could only run on a slice, because the LLM can't do the full pool at all (above). On that controlled, capped head-to-head, Bosun-4B matches it: +12.0 to flash-lite's +12.0, ahead on two of the three tasks. Better data took the 0.6B from +5.3 to +7.0; scale took it the rest of the way to parity.

So the ladder is real, and the choice is yours: Bosun-XS for the cheapest judgment at scale, Bosun-4B when you want frontier-grade quality and the cost/scale advantage in the same model. The LLM, recall, isn't on this chart — it couldn't run the pool.

What we tested, and what we didn't

Bosun is a general judge, and we tried to prove it isn't a one-trick model. We tested it across domains that share nothing but the shape does this satisfy this rule:

• Knowledge-graph warrant — the job it was built for: does the evidence back this connection?
• Instruction-following across relation families — same-topic, shared-entity, cross-domain bridge, and their negations (WarrantBench). It follows the rule, and flips on "not."
• Rules it had never seen — "both mention a figure over $1B," "both involve a regulator." It applied them cold at 0.95, where a frontier LLM managed 0.35. Following some rules taught it to follow new ones.
• Public retrieval — TREC news, expert relevance judgments (FollowIR). State of the art.

We did not benchmark these, but built Bosun for them, and expect it to carry — they are the same symmetric "satisfy-this-rule" shape it already generalized to:

• RAG context grading — keep the chunks that answer, not the ones that merely mention.
• Content moderation — score content against a policy you write as a sentence, and version the policy without retraining.
• Deduplication / entity resolution — are these the same thing, under your definition of "same."
• Agent memory beyond the graph — what to retrieve, retain, and forget, not just how to link it.
• Grounding & citation checks — does this source actually support this claim.

And the honest boundary: every rule Bosun judges today is symmetric. It does not yet do direction — "A precedes B," "A causes B," where argument order matters. We tested that too; it sits at chance, because every relation it has trained on was order-blind. A training gap, not a wall — and it's next.

What's next: Bosun goes to work

We're releasing Bosun-XS (0.6B) and Bosun-4B today. Bosun didn't start as a benchmark model — it started as the graph problem at the top of this post: hold two findings and the edge claimed between them, and ask whether that edge deserves to exist. We have a strong opinion about what makes a connection warranted, and that opinion is the whole reason Bosun exists. But it never went into the weights — it goes in at runtime, as the instruction. Bosun is the engine; the graph is where we hand it ours.

In production, Bosun is the judgment layer of a living graph. Edge by edge, it reads two findings and the connection claimed between them and renders the verdict — under our rule. It doesn't work alone: it's one voice in an ensemble, alongside structural and similarity signals, that together decide what enters the graph, what gets pruned, and what becomes the next thing worth chasing. A graph left to grow drifts into a hairball no model can reason over; the ensemble holds it coherent, and Bosun is the member that weighs the evidence on every edge — against the opinion we bring.

That is exactly why opinion-free matters: the same model that tops FollowIR by executing its opinion of relevance keeps our graph coherent by executing ours of warrant — same weights, different sentence. The benchmark is the proof; the graph is the job; the opinion was always yours to supply.

And the ladder keeps climbing — more rungs, including an 8B.

Some notes on naming

Hanno the Navigator charted coastlines no one had mapped. We took the company's name from him because the work is the same — chart the territory, set the course. But a ship is not run by the navigator alone. The bosun — the boatswain — is the officer who walks the deck and inspects the crew's work, piece by piece, against the standard, and says what passes. Small, hands-on, never the captain; but nothing leaves the deck without the bosun's nod. That is this model: the navigator decides where to go, the bosun makes sure the work that gets done is good. XS is for extra small — a bosun is not a large figure, and that is exactly the point.

Open source

• Model — Bosun-XS weights & inference → · Bosun-4B →
• Evaluation — WarrantBench →

Run it on your own rules. Audit WarrantBench's gold. The lanes where a cheap, steerable, deterministic judge changes what is affordable — RAG filtering, moderation, deduplication, agent memory — are wide open, and this is the model for them.

First of more

Bosun is the first open-source model from Hanno Labs — the AI lab we just launched, building at the intersection of causal intelligence and complex knowledge systems, applying machine intelligence to the problems that still require human judgment. It won't be the last: more models, more research, and more open source are on the way. Follow along at hannolabs.ai.

Appendix A: the rater contract

Prompt:

<|im_start|>system
Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>
<|im_start|>user
<Instruct>: {the rule}
<Query>: {the item}
<Document>: {the evidence}<|im_end|>
<|im_start|>assistant
<think>

</think>

Scoring: score = sigmoid(logit_yes − logit_no) ∈ [0, 1] at the final token. No text generated.

Steering, from the deployed model. The same pair — A: "The SEC proposed new disclosure rules for crypto exchanges." / B: "A startup launched an AI tool for writing marketing emails." — scores 0.05 under "connected only if the same broad topic" and 0.84 under "connected only if different topics." One pair, opposite rules, the answer flips.