"""The capabilities a hosted multimodal model answers, and the runner behind them. Sixteen capabilities in the registry declare an RGB photograph, a closed claim vocabulary and human confirmation, and have no local artefact and no possible one — there is no public dataset for a cattle wound, a Nigerian hoof or a footpad lesion, which is directive §4's whole argument for a visual reasoner here. `app/adapters/multimodal.py` holds the contract and `app/adapters/transports/` the HTTP; this module is what connects them to a capability key so `/jobs` can dispatch on one. ## One runner, sixteen capabilities `claims.schema_for` builds the same envelope for every capability in the registry — `claims`, `observations`, `interpretations`, `confidence`, `evidence`, `limits`, plus `range` and `best_estimate` where the output is ranged. The vocabulary inside it differs per capability and the *shape* does not, so one mapping from that envelope to `InferenceResult` serves all of them and a seventeenth capability is a rubric rather than a module. **What differs per capability is the question**, and that is what `RUBRICS` holds. `SYSTEM_RULES` in `multimodal.py` already carries the safety posture — observation separated from interpretation, no unqualified disease name, no invented field, a null in preference to a guess — so a rubric here says only what to look at and what this particular capability may not conclude. Repeating the general rules in sixteen places is how they drift. ## Why the constraints are written into the rubric and not only the schema The schema refuses a bad answer. The rubric is what stops the model producing one, and for several of these the difference is the product's credibility rather than a validation error: * **`cattle_skin` must never name a disease.** Lumpy skin disease and foot-and-mouth are notifiable under Nigeria's Animal Diseases (Control) Act 1988 s.8(1). A farm acting on an unqualified name from a photograph is a reporting obligation triggered by a guess. * **`cattle_age_dentition` gives a band and never a chronological age.** Teeth carry an eruption stage; a birth year is not in the photograph. * **`cattle_sex` must never block registration.** It is a suggestion beside a field the person fills in. * **`poultry_litter` may describe condition and never moisture.** Caking and friability are visible; water content is not, and it is the number a farm would act on. * **`egg_quality` reads external quality only.** Cracks need candling — RGB cannot see them — and cracks are the valuable half, so the limit is stated rather than left for a farm to discover. * **`poultry_footpad` reports a sampled prevalence, never a house figure.** ## What this module cannot do, and what has to happen instead Everything downstream of the answer checks that it is *well formed*: the schema is closed, the claim is in the vocabulary, the sentence is qualified, the number is in range. **Nothing here can catch an answer that is well formed and wrong.** A hosted model that confidently reads a healthy footpad as grade 3 produces a response that passes every gate in this repository. So the measurement in `experiments/` is not optional polish for these sixteen, it is the control. Until one exists, what stands between the model and a farm is: `experimental` on all sixteen, `requires_review` on every interpretation this runner builds, and `STANDING_WARNING` on every result. **`HUMAN_CONFIRMATION` is not among them, and an earlier draft of this file said it was.** Six of the sixteen carry it — `cattle_age_dentition`, `cattle_bcs`, `cattle_breed`, `cattle_sex`, `poultry_fecal`, `poultry_footpad`. **Ten do not**, and the ten include `cattle_skin` and both wound capabilities: cattle_feces cattle_hoof cattle_skin cattle_wound egg_quality poultry_eye_head poultry_feather poultry_hock poultry_litter poultry_wound The registry's reason is consistent: those ten declare an **observation** output — a description of the picture — and the directive requires confirmation where a capability makes a *judgement*. `requires_review` on every interpretation is that line drawn one level down, and it is the line this runner enforces. Whether an unmeasured hosted model's "description" of a lesion is an observation in that sense is a product question and not this module's to settle. It is written here because the alternative is a reader assuming, as this file once asserted, that all sixteen are behind a confirm button. """ from __future__ import annotations import logging from uuid import UUID, uuid4 from app.adapters import claims from app.adapters.base import AdapterError from app.adapters.claims import ContractViolation from app.adapters.multimodal import HostedMultimodalAdapter from app.adapters.transports import transport_from_env from app.capabilities import Capability from app.media import MediaRef, MediaStore from app.providers import ModelArtefact from app.quality import assess from app.schemas import ( InferenceLocation, InferenceResult, Interpretation, Observation, ) logger = logging.getLogger(__name__) #: What `Capability.model_provider` says for a capability answered over HTTP. #: #: A stable declaration rather than the model's name. The name of the model that #: actually ran is a fact about a *run* and travels on the result — see #: `providers.HostedModelProvider`, which reads it at call time so a rotation is #: a restart rather than a rebuild, and so the release row records which model #: produced which evidence. HOSTED_PROVIDER = "hosted-multimodal" #: The rubric for each capability, keyed by capability key. #: #: Read against each entry's `notes` in `app/capabilities.py`, which carry the #: directive section and the constraint. Where the two could disagree the #: registry wins and this is wrong. RUBRICS: dict[str, str] = { # -- cattle ----------------------------------------------------------- "cattle_bcs": ( "Score the body condition of the animal on the 1-5 scale, where 1 is " "emaciated and 5 is obese. Look at the hooks, the pins, the tail head " "and the ribs. Give a range covering the plausible scores, not a point " "value: a single view of one animal does not support a precise score. " "If the animal is rugged, muddy, heavily haired or standing so the " "hindquarter is hidden, say so in `limits` and widen the range." ), "cattle_age_dentition": ( "Count the permanent incisors that have erupted in the lower jaw and " "give the age BAND that eruption stage indicates. Report the count as " "an observation and the band as the claim.\n\n" "Teeth carry an eruption stage and not a birth year. Never state a " "chronological age, an age in months, or a year of birth — the band is " "the whole answer. If the mouth is not open, the lower incisors are not " "in frame, or wear makes the count ambiguous, set the count null and say " "which in `limits`.\n\n" "**This capability reads teeth and nothing else.** If the lower " "incisors are not visible in the photograph, refuse it — claim " "nothing and choose the limit that says so. Do not fall back on body " "size, frame, horn length or general appearance: those support a guess " "about age and this capability's whole claim is that its answer came " "from dentition. A band derived from the animal's build, published " "under `age_band`, is a different reading wearing this one's name." ), "cattle_breed": ( "Say which breed the animal's phenotype most resembles, from coat " "colour and pattern, horn shape, hump, dewlap, ear carriage and frame.\n\n" "**Not forcing a breed is a required answer and not a failure.** Most " "cattle on a Nigerian smallholding are crossbred, and a crossbred animal " "must be allowed to stay crossbred: where the animal does not sit " "cleanly in one breed, claim `crossbred_or_uncertain` rather than the " "nearest name. Where it does, phrase it as a resemblance — a " "'White-Fulani-like phenotype' — because a photograph supports a " "phenotype and not a pedigree." ), "cattle_sex": ( "Say which sex the animal's visible anatomy suggests — udder, scrotum, " "sheath, frame and horn shape.\n\n" "This is a suggestion beside a field a person fills in, and it must " "never read as settled. If the relevant anatomy is not visible, or the " "animal is young enough that the frame is uninformative, set the claim " "null and say so in `limits`. An unclear answer costs nothing here; a " "confident wrong one is written onto an animal's record." ), "cattle_wound": ( "Describe what is visible at the wound: its approximate size relative to " "the animal, whether the edges are open or closed, whether there is " "discharge, swelling, or fly strike, and how fresh it appears.\n\n" "**Triage only.** There is no public dataset of livestock wounds, so " "this is a description that helps somebody decide whether to look, and " "never a grade, a stage, a healing time or a cause. Never say what " "caused the wound. Where the wound looks deep, contaminated, near a " "joint or eye, or extensive, recommend veterinary review." ), "cattle_skin": ( "Describe the skin: whether there are nodules and roughly how many and " "how widely spread, whether there is hair loss, crusting, scaling, " "thickening or discharge, and which body regions are affected.\n\n" "**Claim what you can actually see, not the vaguest thing that fits.** " "If there are raised nodules, claim `nodular_lesions_visible` — that is " "the finding a farmer can act on. Keep `abnormal_skin_pattern` for skin " "that is clearly wrong in some other way, and do not add it beside the " "nodule claim as padding. Measured on 282 photographs, the nodule claim " "is right 98 times in 100 when it is made and the general one is right " "71; a specific claim is worth more to a farm and it is also the more " "accurate one.\n\n" "**Never name a disease.** Lumpy skin disease and foot-and-mouth are " "notifiable under Nigeria's Animal Diseases (Control) Act 1988 s.8(1), " "and a name from a photograph is a legal obligation triggered by a " "guess. Say what the skin looks like and let the vet name it." ), "cattle_hoof": ( "Describe the hoof: overgrowth of the toe or heel, uneven wear, " "separation at the white line, visible cracks, swelling above the " "coronet, and any discharge or smell of the interdigital space that is " "visible as wetness or matting.\n\n" "A hoof read through mud or dung is a photograph of mud. If the foot is " "not lifted, not washed, or the sole is not in frame, say so in `limits` " "rather than reading through it." ), "cattle_feces": ( "Describe the dropping: its consistency on a firm-to-liquid scale, its " "colour, and whether undigested fibre, mucus, blood or visible worm " "segments are present.\n\n" "Consistency reflects the current diet as much as health, so describe " "what is there and let a person read it against what the animal is " "eating. Never name a parasite species from a photograph of a dropping." ), # -- poultry ---------------------------------------------------------- "poultry_fecal": ( "Describe each dropping visible in the sample: its consistency, its " "colour, and whether blood, mucus, undigested feed or unusually watery " "content is present. Report how many droppings you could actually " "assess as an observation.\n\n" "This is a screen over a sample and never a house diagnosis. Describe " "the droppings and let a person decide; where blood is visible, " "recommend review rather than naming coccidiosis or any other disease." ), "poultry_footpad": ( "Grade the footpad on the 0-4 pododermatitis rubric: 0 is a clean " "unblemished pad, 4 is a large deep lesion with necrosis. Report the " "grade for each bird you can assess and how many birds you assessed.\n\n" "**A sampled prevalence, never a house figure.** The birds in these " "images are the sample; say how many they were and never extrapolate to " "the flock. If a pad is obscured by litter or the bird is not held so " "the pad faces the camera, exclude it and say so." ), "poultry_hock": ( "Describe the hock: whether there is discolouration, a burn, a scab, " "swelling or ulceration, and roughly how severe. Report how many birds " "you assessed.\n\n" "As with the footpad, this is a sample and not a house figure. Hock " "burn and footpad lesions often travel together but are separate " "captures and separate readings; do not infer one from the other." ), "poultry_feather": ( "Describe feather cover over the back and rump: whether coverage is " "complete, thinning or bare, whether there are broken shafts, and " "whether any bare skin shows pecking damage as distinct from moult.\n\n" "Report what the cover looks like. Do not attribute a cause — feather " "loss from pecking, moult, rubbing and nutrition look alike in one " "photograph, and naming a cause is what turns a description into a " "management decision nobody checked." ), "poultry_wound": ( "Describe what is visible at the wound: approximate size, whether the " "skin is broken, whether there is bleeding, scabbing or discharge, and " "whether the surrounding feathers are damaged.\n\n" "**Triage only**, as for cattle. Never state a cause and never a healing " "time. A bird with an open bleeding wound in a populated house needs " "separating, and saying so is the useful output." ), "poultry_eye_head": ( "Describe the head and eyes: whether the eyes are clear or show " "discharge, swelling or closure, whether there is facial or sinus " "swelling, whether the comb and wattles are of normal colour or are " "pale, blue or discoloured, and whether nasal discharge is visible.\n\n" "Comb colour and facial swelling are the two findings a farm acts on " "fastest and the two most often over-read from one photograph. Describe " "them plainly, name no disease, and recommend review where several " "findings appear together." ), "poultry_litter": ( "Describe the litter: whether the surface is loose and friable or caked " "and compacted, whether there is visible capping, and how much of the " "photographed area is affected.\n\n" "**Condition is visible. Moisture is not, and must never be claimed.** " "Caking correlates with moisture and is not a measurement of it, and " "moisture percentage is the number a farm would act on. Describe the " "surface; a farm measures water with a meter." ), "egg_quality": ( "Describe the external quality of the eggs visible: shell cleanliness, " "visible soiling, shape abnormality, ridging or roughness of the shell, " "and obvious breakage. Report how many eggs you could assess.\n\n" "**External quality only, and say so.** A crack that has not broken the " "shell membrane is invisible to a camera and needs candling, and cracks " "are the valuable half of egg quality — so a clean-looking tray here is " "not a tray without cracks. State that limit rather than leaving a farm " "to infer it." ), } #: Said on every run of every capability here, whatever the answer. #: #: Not a hedge for its own sake. These sixteen have no measurement — no #: `experiments/` run has put real photographs through any of them — and a #: hosted model's confident wrong answer is well-formed, so the thing that #: protects a farm is a person looking. `HUMAN_CONFIRMATION` makes that #: structural and this makes it visible. #: Appended to every rubric. The one instruction that applies to all sixteen. #: #: **Written because a measurement showed it was needed.** On `cattle_skin` the #: model reached for the vague claim more than twice as often as the specific #: one, and the specific one was the more accurate of the two — 97.9% against #: 71.4%. A model hedging toward the general claim is not being careful; it is #: producing a sentence a farm cannot act on, and being more wrong while it does. BE_SPECIFIC = ( "Prefer the most specific claim the picture actually supports. A precise " "finding is what a farmer can act on, and a vague one costs them a decision. " "Do not add a general claim beside a specific one as a hedge — if you can " "see the specific thing, claim only that. If you genuinely cannot tell, " "claim nothing and say why in `limits`; that is a better answer than a " "claim nobody can use." ) STANDING_WARNING = ( "Read by an experimental visual model, and not measured on livestock like " "these. Treat it as a second opinion on the photograph and confirm it." ) class ReasoningRunner: """Runs one hosted-multimodal capability against one capture. Constructed per capability key so `RUNNERS` maps a key to a runner exactly as `counting` and `identification` do, and so the rubric is bound at import rather than looked up on every request. """ def __init__(self, capability_key: str, adapter: HostedMultimodalAdapter | None = None): self.capability_key = capability_key self.rubric = RUBRICS[capability_key] # Injectable for the tests, which supply an adapter with a stub # transport. `None` means build one from the environment at call time, # which is what keeps a key rotation a restart. self._adapter = adapter def _vocabulary_note(self, capability: Capability) -> str: """The words and array bounds the schema knows and cannot send. ## Why this exists at all Two controls in `claims.schema_for` do not survive the trip to a provider, and the model then breaks them in the obvious way: * **`ClaimQuantity.values`.** `check_observation_values` refuses an observation whose `value` is a word the claim did not declare, and the schema has nowhere to put those words — `value` is `number | string | null`, and structured outputs refuses an `enum` beside a union type. So the model is told `body_condition_band` may carry a word and never which words, and the first real call answered `"Appropriate"` and then `"Thin to Appropriate"`. Both are refused, both are the model doing its best with what it was given, and both cost a vendor call. * **`maxItems`.** `_for_structured_output` strips it because the API refuses it, which leaves every array unbounded. The first real call also produced an answer truncated at the token ceiling — a runaway list — and a truncated answer is a whole capture refused. `claims.enforce` still catches both, so this is not a control. It is what stops the model failing a control it was never shown, which is the difference between a capability that works and one that retries. **Generated from the registry, never typed.** A word list written here would be a second copy of `ClaimQuantity.values` and would drift from the check that enforces it. """ contract = claims.schema_for(capability) output = capability.acquisition.output lines: list[str] = [] exact, named, wordless = [], [], [] for quantity in (output.quantities or ()): permitted, free_reason = output.words_for(quantity.claim) if free_reason is not None: named.append(quantity.claim) elif permitted: offered = ", ".join(f'"{w}"' for w in permitted) exact.append(f" - {quantity.claim}: {offered}") else: wordless.append(quantity.claim) if exact: lines.append( "Where an observation's `value` is a word it must be exactly " "one of the words listed for that claim — lower case, as " "written, not a phrase and not a pair joined by 'to'. If the " "picture sits between two of them choose the nearer one and " "widen the range instead.\n" + "\n".join(exact) ) if named: lines.append( f"These claims carry a NAME and nothing else: " f"{', '.join(named)}. A name is at most {claims._NAME_WORDS} " f"words and {claims._NAME_LENGTH} characters, with no digit and " f"no punctuation beyond a hyphen or an apostrophe — " f'"White Fulani", not a sentence describing one. Put the ' f"reasoning in `evidence`, which is a closed list, and never " f"here." ) if wordless: lines.append( f"These claims carry NO word: {', '.join(wordless)}. The claim " f"itself is the finding. Their `value` is a number or null, " f"never text." ) bounds = [] for field, spec in (contract.get("properties") or {}).items(): ceiling = spec.get("maxItems") if isinstance(spec, dict) else None if isinstance(ceiling, int): bounds.append(f"{field} at most {ceiling}") if bounds: lines.append( "Length limits, which the schema cannot express and which are " "checked after you answer: " + "; ".join(bounds) + "." ) if output.show_range: lines.append( "`range` is two DIFFERENT numbers, low then high. A pair like " "[0, 0] is an exact figure wearing a range's clothes and the " "whole answer is discarded for it.\n\n" "If what this capability measures is NOT VISIBLE in the " "picture, refuse: choose the limit that says the subject is not " "in frame, claim nothing, and let a person take it again. Do " "not answer with zeros, and do not answer with the widest band " "the scale allows either — a band from one end of the scale to " "the other is not a cautious reading, it is a refusal a farmer " "has to decode, and it reaches them looking like a measurement. " "Widen a band because the picture is poor; refuse because the " "subject is absent. They are different answers." ) return "\n\n".join(lines) def _reasoner(self) -> HostedMultimodalAdapter: if self._adapter is not None: return self._adapter return HostedMultimodalAdapter(transport=transport_from_env()).load() def run( self, *, request, capability: Capability, artefact: ModelArtefact, store: MediaStore, request_id: UUID | None = None, ) -> InferenceResult: request_id = request_id or uuid4() warnings: list[str] = [STANDING_WARNING] images = [] for media_id in request.media_ids: images.append(store.open_image(MediaRef( media_id=media_id, farm_id=request.farm_id, captured_at=request.captured_at, object_path=request.path_for(media_id), ))) # **The quality gate runs before the call, not after it.** A blurred or # dark frame is refused here for free; sending it costs a vendor call # and returns a confident reading of a photograph nobody could read. verdict = assess(images[0]) checks = list(verdict.checks) if verdict.blocked: return self._result( request=request, capability=capability, artefact=artefact, request_id=request_id, observations=[], interpretations=[], confidence=None, checks=checks, warnings=warnings + [ "The photograph was not clear enough to read, so nothing was " "sent to the model and nothing is claimed." ], recapture=True, ) try: note = self._vocabulary_note(capability) rubric = f"{self.rubric}\n\n{BE_SPECIFIC}" response = self._reasoner().reason( images, capability=capability, rubric=f"{rubric}\n\n{note}" if note else rubric, ) except ContractViolation as exc: # **Answered, and the answer may not be published.** Distinct from # the transport failure below, and the distinction decides whether # the run is retried. # # The model produced something; it broke the schema, the vocabulary, # a numeric bound or the forbidden-claims gate. Re-asking the same # photograph asks the same question of the same model and gets the # same answer, so raising here would queue a capture that fails # identically for ever — `MAX_ATTEMPTS` eventually stops it, at the # cost of that many vendor calls. # # The commonest cause is an out-of-protocol capture. `cattle_wound` # is a `lesion_close_up`; photograph a whole herd with it and the # honest answer is that there is no wound, which the contract has no # way to express — §7 requires a band from a capability that # declares one, and `[0, 0]` is an exact figure wearing a band's # clothes. Both rules are right and together they leave the model # nowhere to stand, so the capture is refused rather than the rules # bent. # # A farm gets a recapture prompt, nothing is claimed, and the reason # is on the run. logger.info( "%s could not be published for %s: %s", capability.key, request.media_ids[0], exc, ) return self._result( request=request, capability=capability, artefact=artefact, request_id=request_id, observations=[], interpretations=[], confidence=None, checks=checks, warnings=warnings + [ "This photograph did not support a reading this capability " "may publish, so nothing is claimed for it. Take it again " "the way the capture screen describes.", ], recapture=True, ) except AdapterError as exc: # **Not answered.** A timeout, a refused key, a 5xx, a truncated # body. `/jobs` turns a ValueError into a FAILED job which the API # keeps queued, and that is right here: the question was never put, # so asking it again is a different attempt rather than the same # one. raise ValueError(str(exc)) from exc warnings.extend(response.warnings) return self._from_envelope( parsed=response.parsed, request=request, capability=capability, artefact=artefact, request_id=request_id, checks=checks, warnings=warnings, ) def _from_envelope( self, *, parsed: dict, request, capability, artefact, request_id, checks, warnings, ) -> InferenceResult: """`claims.schema_for`'s envelope as an `InferenceResult`. Everything here has already been through `claims.parse_strict` and `claims.enforce`, so the vocabulary, the bounds and the forbidden-claim gate have all run. What is left is a transcription. """ observations = [ Observation( type=str(item["type"]), value=item.get("value"), unit=item.get("unit"), confidence=item.get("confidence"), ) for item in (parsed.get("observations") or []) if isinstance(item, dict) and item.get("type") ] # A ranged capability carries its answer in `range`, and the range is # the answer — `best_estimate` is emitted beside it and never instead of # it, because §7's whole complaint is a point value standing in for a # band. band = parsed.get("range") if isinstance(band, list) and len(band) == 2: output_type = capability.acquisition.output.type observations.append(Observation( type=output_type, value=float(band[0]), unit="range_low", confidence=None, )) observations.append(Observation( type=output_type, value=float(band[1]), unit="range_high", confidence=None, )) interpretations = [ Interpretation( label=str(item["label"]), confidence=item.get("confidence"), # Always true, and the schema pins it to `const: true`. A # hosted reading a person has not seen is not a record. requires_review=True, ) for item in (parsed.get("interpretations") or []) if isinstance(item, dict) and item.get("label") ] # The closed lists travel as warnings rather than as claims. They are # what the model was allowed to say *about its own answer*, and a farm # reading "One view only" beside a band is the point of them. for limit in (parsed.get("limits") or []): if isinstance(limit, str) and limit.strip(): warnings.append(limit.strip()) return self._result( request=request, capability=capability, artefact=artefact, request_id=request_id, observations=observations, interpretations=interpretations, confidence=parsed.get("confidence"), checks=checks, warnings=warnings, recapture=False, ) def _result( self, *, request, capability, artefact, request_id, observations, interpretations, confidence, checks, warnings, recapture, ) -> InferenceResult: banned = set(capability.acquisition.forbidden_claims) offending = sorted( {o.type for o in observations if o.type in banned} | {i.label for i in interpretations if i.label in banned} ) if offending: # `claims.enforce` already refuses these, and this refuses them # again on the way out. The gate that matters is the one closest to # the thing being protected, and a runner is the last place a # forbidden claim could be introduced. raise ValueError( f"{capability.key} tried to emit {offending}, which its own " f"registry entry forbids by name." ) return InferenceResult( request_id=request_id, capability_key=capability.key, model_id=artefact.model_id, model_version=artefact.version, inference_location=InferenceLocation.REMOTE, subject_type=request.subject_type, subject_id=request.subject_id, observations=observations, interpretations=interpretations, observation_confidence=confidence, # **Never set, on any path.** The three-level label beside an # interpretation would read as a confidence in the interpretation, # and nothing has calibrated one — these sixteen have no measurement # at all. `requires_review` is the honest field and it is always # true. interpretation_confidence=None, quality_checks=checks, warnings=warnings, recommended_recapture=recapture, ) #: Every capability a hosted multimodal model answers. #: #: Derived from `RUBRICS` rather than listed twice: a capability with a rubric #: and no runner would be silently unavailable, and a runner with no rubric #: cannot be constructed. RUNNERS: dict[str, ReasoningRunner] = { key: ReasoningRunner(key) for key in RUBRICS }