animap-gpu / app /reasoning.py
bluman1's picture
Publish services/inference
4b98524 verified
Raw
History Blame Contribute Delete
33.1 kB
"""The capabilities a hosted multimodal model answers, and the runner behind them.
Sixteen capabilities in the registry declare an RGB photograph, a closed claim
vocabulary and human confirmation, and have no local artefact and no possible
one — there is no public dataset for a cattle wound, a Nigerian hoof or a
footpad lesion, which is directive §4's whole argument for a visual reasoner
here. `app/adapters/multimodal.py` holds the contract and
`app/adapters/transports/` the HTTP; this module is what connects them to a
capability key so `/jobs` can dispatch on one.
## One runner, sixteen capabilities
`claims.schema_for` builds the same envelope for every capability in the
registry — `claims`, `observations`, `interpretations`, `confidence`,
`evidence`, `limits`, plus `range` and `best_estimate` where the output is
ranged. The vocabulary inside it differs per capability and the *shape* does
not, so one mapping from that envelope to `InferenceResult` serves all of them
and a seventeenth capability is a rubric rather than a module.
**What differs per capability is the question**, and that is what `RUBRICS`
holds. `SYSTEM_RULES` in `multimodal.py` already carries the safety posture —
observation separated from interpretation, no unqualified disease name, no
invented field, a null in preference to a guess — so a rubric here says only
what to look at and what this particular capability may not conclude. Repeating
the general rules in sixteen places is how they drift.
## Why the constraints are written into the rubric and not only the schema
The schema refuses a bad answer. The rubric is what stops the model producing
one, and for several of these the difference is the product's credibility rather
than a validation error:
* **`cattle_skin` must never name a disease.** Lumpy skin disease and
foot-and-mouth are notifiable under Nigeria's Animal Diseases (Control) Act
1988 s.8(1). A farm acting on an unqualified name from a photograph is a
reporting obligation triggered by a guess.
* **`cattle_age_dentition` gives a band and never a chronological age.**
Teeth carry an eruption stage; a birth year is not in the photograph.
* **`cattle_sex` must never block registration.** It is a suggestion beside a
field the person fills in.
* **`poultry_litter` may describe condition and never moisture.** Caking and
friability are visible; water content is not, and it is the number a farm
would act on.
* **`egg_quality` reads external quality only.** Cracks need candling — RGB
cannot see them — and cracks are the valuable half, so the limit is stated
rather than left for a farm to discover.
* **`poultry_footpad` reports a sampled prevalence, never a house figure.**
## What this module cannot do, and what has to happen instead
Everything downstream of the answer checks that it is *well formed*: the schema
is closed, the claim is in the vocabulary, the sentence is qualified, the number
is in range. **Nothing here can catch an answer that is well formed and wrong.**
A hosted model that confidently reads a healthy footpad as grade 3 produces a
response that passes every gate in this repository.
So the measurement in `experiments/` is not optional polish for these sixteen,
it is the control. Until one exists, what stands between the model and a farm
is: `experimental` on all sixteen, `requires_review` on every interpretation
this runner builds, and `STANDING_WARNING` on every result.
**`HUMAN_CONFIRMATION` is not among them, and an earlier draft of this file said
it was.** Six of the sixteen carry it — `cattle_age_dentition`, `cattle_bcs`,
`cattle_breed`, `cattle_sex`, `poultry_fecal`, `poultry_footpad`. **Ten do
not**, and the ten include `cattle_skin` and both wound capabilities:
cattle_feces cattle_hoof cattle_skin cattle_wound
egg_quality poultry_eye_head poultry_feather poultry_hock
poultry_litter poultry_wound
The registry's reason is consistent: those ten declare an **observation** output
— a description of the picture — and the directive requires confirmation where a
capability makes a *judgement*. `requires_review` on every interpretation is that
line drawn one level down, and it is the line this runner enforces.
Whether an unmeasured hosted model's "description" of a lesion is an observation
in that sense is a product question and not this module's to settle. It is
written here because the alternative is a reader assuming, as this file once
asserted, that all sixteen are behind a confirm button.
"""
from __future__ import annotations
import logging
from uuid import UUID, uuid4
from app.adapters import claims
from app.adapters.base import AdapterError
from app.adapters.claims import ContractViolation
from app.adapters.multimodal import HostedMultimodalAdapter
from app.adapters.transports import transport_from_env
from app.capabilities import Capability
from app.media import MediaRef, MediaStore
from app.providers import ModelArtefact
from app.quality import assess
from app.schemas import (
InferenceLocation,
InferenceResult,
Interpretation,
Observation,
)
logger = logging.getLogger(__name__)
#: What `Capability.model_provider` says for a capability answered over HTTP.
#:
#: A stable declaration rather than the model's name. The name of the model that
#: actually ran is a fact about a *run* and travels on the result — see
#: `providers.HostedModelProvider`, which reads it at call time so a rotation is
#: a restart rather than a rebuild, and so the release row records which model
#: produced which evidence.
HOSTED_PROVIDER = "hosted-multimodal"
#: The rubric for each capability, keyed by capability key.
#:
#: Read against each entry's `notes` in `app/capabilities.py`, which carry the
#: directive section and the constraint. Where the two could disagree the
#: registry wins and this is wrong.
RUBRICS: dict[str, str] = {
# -- cattle -----------------------------------------------------------
"cattle_bcs": (
"Score the body condition of the animal on the 1-5 scale, where 1 is "
"emaciated and 5 is obese. Look at the hooks, the pins, the tail head "
"and the ribs. Give a range covering the plausible scores, not a point "
"value: a single view of one animal does not support a precise score. "
"If the animal is rugged, muddy, heavily haired or standing so the "
"hindquarter is hidden, say so in `limits` and widen the range."
),
"cattle_age_dentition": (
"Count the permanent incisors that have erupted in the lower jaw and "
"give the age BAND that eruption stage indicates. Report the count as "
"an observation and the band as the claim.\n\n"
"Teeth carry an eruption stage and not a birth year. Never state a "
"chronological age, an age in months, or a year of birth — the band is "
"the whole answer. If the mouth is not open, the lower incisors are not "
"in frame, or wear makes the count ambiguous, set the count null and say "
"which in `limits`.\n\n"
"**This capability reads teeth and nothing else.** If the lower "
"incisors are not visible in the photograph, refuse it — claim "
"nothing and choose the limit that says so. Do not fall back on body "
"size, frame, horn length or general appearance: those support a guess "
"about age and this capability's whole claim is that its answer came "
"from dentition. A band derived from the animal's build, published "
"under `age_band`, is a different reading wearing this one's name."
),
"cattle_breed": (
"Say which breed the animal's phenotype most resembles, from coat "
"colour and pattern, horn shape, hump, dewlap, ear carriage and frame.\n\n"
"**Not forcing a breed is a required answer and not a failure.** Most "
"cattle on a Nigerian smallholding are crossbred, and a crossbred animal "
"must be allowed to stay crossbred: where the animal does not sit "
"cleanly in one breed, claim `crossbred_or_uncertain` rather than the "
"nearest name. Where it does, phrase it as a resemblance — a "
"'White-Fulani-like phenotype' — because a photograph supports a "
"phenotype and not a pedigree."
),
"cattle_sex": (
"Say which sex the animal's visible anatomy suggests — udder, scrotum, "
"sheath, frame and horn shape.\n\n"
"This is a suggestion beside a field a person fills in, and it must "
"never read as settled. If the relevant anatomy is not visible, or the "
"animal is young enough that the frame is uninformative, set the claim "
"null and say so in `limits`. An unclear answer costs nothing here; a "
"confident wrong one is written onto an animal's record."
),
"cattle_wound": (
"Describe what is visible at the wound: its approximate size relative to "
"the animal, whether the edges are open or closed, whether there is "
"discharge, swelling, or fly strike, and how fresh it appears.\n\n"
"**Triage only.** There is no public dataset of livestock wounds, so "
"this is a description that helps somebody decide whether to look, and "
"never a grade, a stage, a healing time or a cause. Never say what "
"caused the wound. Where the wound looks deep, contaminated, near a "
"joint or eye, or extensive, recommend veterinary review."
),
"cattle_skin": (
"Describe the skin: whether there are nodules and roughly how many and "
"how widely spread, whether there is hair loss, crusting, scaling, "
"thickening or discharge, and which body regions are affected.\n\n"
"**Claim what you can actually see, not the vaguest thing that fits.** "
"If there are raised nodules, claim `nodular_lesions_visible` — that is "
"the finding a farmer can act on. Keep `abnormal_skin_pattern` for skin "
"that is clearly wrong in some other way, and do not add it beside the "
"nodule claim as padding. Measured on 282 photographs, the nodule claim "
"is right 98 times in 100 when it is made and the general one is right "
"71; a specific claim is worth more to a farm and it is also the more "
"accurate one.\n\n"
"**Never name a disease.** Lumpy skin disease and foot-and-mouth are "
"notifiable under Nigeria's Animal Diseases (Control) Act 1988 s.8(1), "
"and a name from a photograph is a legal obligation triggered by a "
"guess. Say what the skin looks like and let the vet name it."
),
"cattle_hoof": (
"Describe the hoof: overgrowth of the toe or heel, uneven wear, "
"separation at the white line, visible cracks, swelling above the "
"coronet, and any discharge or smell of the interdigital space that is "
"visible as wetness or matting.\n\n"
"A hoof read through mud or dung is a photograph of mud. If the foot is "
"not lifted, not washed, or the sole is not in frame, say so in `limits` "
"rather than reading through it."
),
"cattle_feces": (
"Describe the dropping: its consistency on a firm-to-liquid scale, its "
"colour, and whether undigested fibre, mucus, blood or visible worm "
"segments are present.\n\n"
"Consistency reflects the current diet as much as health, so describe "
"what is there and let a person read it against what the animal is "
"eating. Never name a parasite species from a photograph of a dropping."
),
# -- poultry ----------------------------------------------------------
"poultry_fecal": (
"Describe each dropping visible in the sample: its consistency, its "
"colour, and whether blood, mucus, undigested feed or unusually watery "
"content is present. Report how many droppings you could actually "
"assess as an observation.\n\n"
"This is a screen over a sample and never a house diagnosis. Describe "
"the droppings and let a person decide; where blood is visible, "
"recommend review rather than naming coccidiosis or any other disease."
),
"poultry_footpad": (
"Grade the footpad on the 0-4 pododermatitis rubric: 0 is a clean "
"unblemished pad, 4 is a large deep lesion with necrosis. Report the "
"grade for each bird you can assess and how many birds you assessed.\n\n"
"**A sampled prevalence, never a house figure.** The birds in these "
"images are the sample; say how many they were and never extrapolate to "
"the flock. If a pad is obscured by litter or the bird is not held so "
"the pad faces the camera, exclude it and say so."
),
"poultry_hock": (
"Describe the hock: whether there is discolouration, a burn, a scab, "
"swelling or ulceration, and roughly how severe. Report how many birds "
"you assessed.\n\n"
"As with the footpad, this is a sample and not a house figure. Hock "
"burn and footpad lesions often travel together but are separate "
"captures and separate readings; do not infer one from the other."
),
"poultry_feather": (
"Describe feather cover over the back and rump: whether coverage is "
"complete, thinning or bare, whether there are broken shafts, and "
"whether any bare skin shows pecking damage as distinct from moult.\n\n"
"Report what the cover looks like. Do not attribute a cause — feather "
"loss from pecking, moult, rubbing and nutrition look alike in one "
"photograph, and naming a cause is what turns a description into a "
"management decision nobody checked."
),
"poultry_wound": (
"Describe what is visible at the wound: approximate size, whether the "
"skin is broken, whether there is bleeding, scabbing or discharge, and "
"whether the surrounding feathers are damaged.\n\n"
"**Triage only**, as for cattle. Never state a cause and never a healing "
"time. A bird with an open bleeding wound in a populated house needs "
"separating, and saying so is the useful output."
),
"poultry_eye_head": (
"Describe the head and eyes: whether the eyes are clear or show "
"discharge, swelling or closure, whether there is facial or sinus "
"swelling, whether the comb and wattles are of normal colour or are "
"pale, blue or discoloured, and whether nasal discharge is visible.\n\n"
"Comb colour and facial swelling are the two findings a farm acts on "
"fastest and the two most often over-read from one photograph. Describe "
"them plainly, name no disease, and recommend review where several "
"findings appear together."
),
"poultry_litter": (
"Describe the litter: whether the surface is loose and friable or caked "
"and compacted, whether there is visible capping, and how much of the "
"photographed area is affected.\n\n"
"**Condition is visible. Moisture is not, and must never be claimed.** "
"Caking correlates with moisture and is not a measurement of it, and "
"moisture percentage is the number a farm would act on. Describe the "
"surface; a farm measures water with a meter."
),
"egg_quality": (
"Describe the external quality of the eggs visible: shell cleanliness, "
"visible soiling, shape abnormality, ridging or roughness of the shell, "
"and obvious breakage. Report how many eggs you could assess.\n\n"
"**External quality only, and say so.** A crack that has not broken the "
"shell membrane is invisible to a camera and needs candling, and cracks "
"are the valuable half of egg quality — so a clean-looking tray here is "
"not a tray without cracks. State that limit rather than leaving a farm "
"to infer it."
),
}
#: Said on every run of every capability here, whatever the answer.
#:
#: Not a hedge for its own sake. These sixteen have no measurement — no
#: `experiments/` run has put real photographs through any of them — and a
#: hosted model's confident wrong answer is well-formed, so the thing that
#: protects a farm is a person looking. `HUMAN_CONFIRMATION` makes that
#: structural and this makes it visible.
#: Appended to every rubric. The one instruction that applies to all sixteen.
#:
#: **Written because a measurement showed it was needed.** On `cattle_skin` the
#: model reached for the vague claim more than twice as often as the specific
#: one, and the specific one was the more accurate of the two — 97.9% against
#: 71.4%. A model hedging toward the general claim is not being careful; it is
#: producing a sentence a farm cannot act on, and being more wrong while it does.
BE_SPECIFIC = (
"Prefer the most specific claim the picture actually supports. A precise "
"finding is what a farmer can act on, and a vague one costs them a decision. "
"Do not add a general claim beside a specific one as a hedge — if you can "
"see the specific thing, claim only that. If you genuinely cannot tell, "
"claim nothing and say why in `limits`; that is a better answer than a "
"claim nobody can use."
)
STANDING_WARNING = (
"Read by an experimental visual model, and not measured on livestock like "
"these. Treat it as a second opinion on the photograph and confirm it."
)
class ReasoningRunner:
"""Runs one hosted-multimodal capability against one capture.
Constructed per capability key so `RUNNERS` maps a key to a runner exactly
as `counting` and `identification` do, and so the rubric is bound at import
rather than looked up on every request.
"""
def __init__(self, capability_key: str, adapter: HostedMultimodalAdapter | None = None):
self.capability_key = capability_key
self.rubric = RUBRICS[capability_key]
# Injectable for the tests, which supply an adapter with a stub
# transport. `None` means build one from the environment at call time,
# which is what keeps a key rotation a restart.
self._adapter = adapter
def _vocabulary_note(self, capability: Capability) -> str:
"""The words and array bounds the schema knows and cannot send.
## Why this exists at all
Two controls in `claims.schema_for` do not survive the trip to a
provider, and the model then breaks them in the obvious way:
* **`ClaimQuantity.values`.** `check_observation_values` refuses an
observation whose `value` is a word the claim did not declare, and
the schema has nowhere to put those words — `value` is
`number | string | null`, and structured outputs refuses an `enum`
beside a union type. So the model is told `body_condition_band` may
carry a word and never which words, and the first real call answered
`"Appropriate"` and then `"Thin to Appropriate"`. Both are refused,
both are the model doing its best with what it was given, and both
cost a vendor call.
* **`maxItems`.** `_for_structured_output` strips it because the API
refuses it, which leaves every array unbounded. The first real call
also produced an answer truncated at the token ceiling — a runaway
list — and a truncated answer is a whole capture refused.
`claims.enforce` still catches both, so this is not a control. It is what
stops the model failing a control it was never shown, which is the
difference between a capability that works and one that retries.
**Generated from the registry, never typed.** A word list written here
would be a second copy of `ClaimQuantity.values` and would drift from
the check that enforces it.
"""
contract = claims.schema_for(capability)
output = capability.acquisition.output
lines: list[str] = []
exact, named, wordless = [], [], []
for quantity in (output.quantities or ()):
permitted, free_reason = output.words_for(quantity.claim)
if free_reason is not None:
named.append(quantity.claim)
elif permitted:
offered = ", ".join(f'"{w}"' for w in permitted)
exact.append(f" - {quantity.claim}: {offered}")
else:
wordless.append(quantity.claim)
if exact:
lines.append(
"Where an observation's `value` is a word it must be exactly "
"one of the words listed for that claim — lower case, as "
"written, not a phrase and not a pair joined by 'to'. If the "
"picture sits between two of them choose the nearer one and "
"widen the range instead.\n" + "\n".join(exact)
)
if named:
lines.append(
f"These claims carry a NAME and nothing else: "
f"{', '.join(named)}. A name is at most {claims._NAME_WORDS} "
f"words and {claims._NAME_LENGTH} characters, with no digit and "
f"no punctuation beyond a hyphen or an apostrophe — "
f'"White Fulani", not a sentence describing one. Put the '
f"reasoning in `evidence`, which is a closed list, and never "
f"here."
)
if wordless:
lines.append(
f"These claims carry NO word: {', '.join(wordless)}. The claim "
f"itself is the finding. Their `value` is a number or null, "
f"never text."
)
bounds = []
for field, spec in (contract.get("properties") or {}).items():
ceiling = spec.get("maxItems") if isinstance(spec, dict) else None
if isinstance(ceiling, int):
bounds.append(f"{field} at most {ceiling}")
if bounds:
lines.append(
"Length limits, which the schema cannot express and which are "
"checked after you answer: " + "; ".join(bounds) + "."
)
if output.show_range:
lines.append(
"`range` is two DIFFERENT numbers, low then high. A pair like "
"[0, 0] is an exact figure wearing a range's clothes and the "
"whole answer is discarded for it.\n\n"
"If what this capability measures is NOT VISIBLE in the "
"picture, refuse: choose the limit that says the subject is not "
"in frame, claim nothing, and let a person take it again. Do "
"not answer with zeros, and do not answer with the widest band "
"the scale allows either — a band from one end of the scale to "
"the other is not a cautious reading, it is a refusal a farmer "
"has to decode, and it reaches them looking like a measurement. "
"Widen a band because the picture is poor; refuse because the "
"subject is absent. They are different answers."
)
return "\n\n".join(lines)
def _reasoner(self) -> HostedMultimodalAdapter:
if self._adapter is not None:
return self._adapter
return HostedMultimodalAdapter(transport=transport_from_env()).load()
def run(
self,
*,
request,
capability: Capability,
artefact: ModelArtefact,
store: MediaStore,
request_id: UUID | None = None,
) -> InferenceResult:
request_id = request_id or uuid4()
warnings: list[str] = [STANDING_WARNING]
images = []
for media_id in request.media_ids:
images.append(store.open_image(MediaRef(
media_id=media_id,
farm_id=request.farm_id,
captured_at=request.captured_at,
object_path=request.path_for(media_id),
)))
# **The quality gate runs before the call, not after it.** A blurred or
# dark frame is refused here for free; sending it costs a vendor call
# and returns a confident reading of a photograph nobody could read.
verdict = assess(images[0])
checks = list(verdict.checks)
if verdict.blocked:
return self._result(
request=request, capability=capability, artefact=artefact,
request_id=request_id, observations=[], interpretations=[],
confidence=None, checks=checks,
warnings=warnings + [
"The photograph was not clear enough to read, so nothing was "
"sent to the model and nothing is claimed."
],
recapture=True,
)
try:
note = self._vocabulary_note(capability)
rubric = f"{self.rubric}\n\n{BE_SPECIFIC}"
response = self._reasoner().reason(
images, capability=capability,
rubric=f"{rubric}\n\n{note}" if note else rubric,
)
except ContractViolation as exc:
# **Answered, and the answer may not be published.** Distinct from
# the transport failure below, and the distinction decides whether
# the run is retried.
#
# The model produced something; it broke the schema, the vocabulary,
# a numeric bound or the forbidden-claims gate. Re-asking the same
# photograph asks the same question of the same model and gets the
# same answer, so raising here would queue a capture that fails
# identically for ever — `MAX_ATTEMPTS` eventually stops it, at the
# cost of that many vendor calls.
#
# The commonest cause is an out-of-protocol capture. `cattle_wound`
# is a `lesion_close_up`; photograph a whole herd with it and the
# honest answer is that there is no wound, which the contract has no
# way to express — §7 requires a band from a capability that
# declares one, and `[0, 0]` is an exact figure wearing a band's
# clothes. Both rules are right and together they leave the model
# nowhere to stand, so the capture is refused rather than the rules
# bent.
#
# A farm gets a recapture prompt, nothing is claimed, and the reason
# is on the run.
logger.info(
"%s could not be published for %s: %s",
capability.key, request.media_ids[0], exc,
)
return self._result(
request=request, capability=capability, artefact=artefact,
request_id=request_id, observations=[], interpretations=[],
confidence=None, checks=checks,
warnings=warnings + [
"This photograph did not support a reading this capability "
"may publish, so nothing is claimed for it. Take it again "
"the way the capture screen describes.",
],
recapture=True,
)
except AdapterError as exc:
# **Not answered.** A timeout, a refused key, a 5xx, a truncated
# body. `/jobs` turns a ValueError into a FAILED job which the API
# keeps queued, and that is right here: the question was never put,
# so asking it again is a different attempt rather than the same
# one.
raise ValueError(str(exc)) from exc
warnings.extend(response.warnings)
return self._from_envelope(
parsed=response.parsed, request=request, capability=capability,
artefact=artefact, request_id=request_id, checks=checks,
warnings=warnings,
)
def _from_envelope(
self, *, parsed: dict, request, capability, artefact, request_id,
checks, warnings,
) -> InferenceResult:
"""`claims.schema_for`'s envelope as an `InferenceResult`.
Everything here has already been through `claims.parse_strict` and
`claims.enforce`, so the vocabulary, the bounds and the forbidden-claim
gate have all run. What is left is a transcription.
"""
observations = [
Observation(
type=str(item["type"]),
value=item.get("value"),
unit=item.get("unit"),
confidence=item.get("confidence"),
)
for item in (parsed.get("observations") or [])
if isinstance(item, dict) and item.get("type")
]
# A ranged capability carries its answer in `range`, and the range is
# the answer — `best_estimate` is emitted beside it and never instead of
# it, because §7's whole complaint is a point value standing in for a
# band.
band = parsed.get("range")
if isinstance(band, list) and len(band) == 2:
output_type = capability.acquisition.output.type
observations.append(Observation(
type=output_type, value=float(band[0]), unit="range_low",
confidence=None,
))
observations.append(Observation(
type=output_type, value=float(band[1]), unit="range_high",
confidence=None,
))
interpretations = [
Interpretation(
label=str(item["label"]),
confidence=item.get("confidence"),
# Always true, and the schema pins it to `const: true`. A
# hosted reading a person has not seen is not a record.
requires_review=True,
)
for item in (parsed.get("interpretations") or [])
if isinstance(item, dict) and item.get("label")
]
# The closed lists travel as warnings rather than as claims. They are
# what the model was allowed to say *about its own answer*, and a farm
# reading "One view only" beside a band is the point of them.
for limit in (parsed.get("limits") or []):
if isinstance(limit, str) and limit.strip():
warnings.append(limit.strip())
return self._result(
request=request, capability=capability, artefact=artefact,
request_id=request_id, observations=observations,
interpretations=interpretations,
confidence=parsed.get("confidence"),
checks=checks, warnings=warnings, recapture=False,
)
def _result(
self, *, request, capability, artefact, request_id, observations,
interpretations, confidence, checks, warnings, recapture,
) -> InferenceResult:
banned = set(capability.acquisition.forbidden_claims)
offending = sorted(
{o.type for o in observations if o.type in banned}
| {i.label for i in interpretations if i.label in banned}
)
if offending:
# `claims.enforce` already refuses these, and this refuses them
# again on the way out. The gate that matters is the one closest to
# the thing being protected, and a runner is the last place a
# forbidden claim could be introduced.
raise ValueError(
f"{capability.key} tried to emit {offending}, which its own "
f"registry entry forbids by name."
)
return InferenceResult(
request_id=request_id,
capability_key=capability.key,
model_id=artefact.model_id,
model_version=artefact.version,
inference_location=InferenceLocation.REMOTE,
subject_type=request.subject_type,
subject_id=request.subject_id,
observations=observations,
interpretations=interpretations,
observation_confidence=confidence,
# **Never set, on any path.** The three-level label beside an
# interpretation would read as a confidence in the interpretation,
# and nothing has calibrated one — these sixteen have no measurement
# at all. `requires_review` is the honest field and it is always
# true.
interpretation_confidence=None,
quality_checks=checks,
warnings=warnings,
recommended_recapture=recapture,
)
#: Every capability a hosted multimodal model answers.
#:
#: Derived from `RUBRICS` rather than listed twice: a capability with a rubric
#: and no runner would be silently unavailable, and a runner with no rubric
#: cannot be constructed.
RUNNERS: dict[str, ReasoningRunner] = {
key: ReasoningRunner(key) for key in RUBRICS
}