Spaces:
Running on Zero
Running on Zero
| """The capabilities a hosted multimodal model answers, and the runner behind them. | |
| Sixteen capabilities in the registry declare an RGB photograph, a closed claim | |
| vocabulary and human confirmation, and have no local artefact and no possible | |
| one — there is no public dataset for a cattle wound, a Nigerian hoof or a | |
| footpad lesion, which is directive §4's whole argument for a visual reasoner | |
| here. `app/adapters/multimodal.py` holds the contract and | |
| `app/adapters/transports/` the HTTP; this module is what connects them to a | |
| capability key so `/jobs` can dispatch on one. | |
| ## One runner, sixteen capabilities | |
| `claims.schema_for` builds the same envelope for every capability in the | |
| registry — `claims`, `observations`, `interpretations`, `confidence`, | |
| `evidence`, `limits`, plus `range` and `best_estimate` where the output is | |
| ranged. The vocabulary inside it differs per capability and the *shape* does | |
| not, so one mapping from that envelope to `InferenceResult` serves all of them | |
| and a seventeenth capability is a rubric rather than a module. | |
| **What differs per capability is the question**, and that is what `RUBRICS` | |
| holds. `SYSTEM_RULES` in `multimodal.py` already carries the safety posture — | |
| observation separated from interpretation, no unqualified disease name, no | |
| invented field, a null in preference to a guess — so a rubric here says only | |
| what to look at and what this particular capability may not conclude. Repeating | |
| the general rules in sixteen places is how they drift. | |
| ## Why the constraints are written into the rubric and not only the schema | |
| The schema refuses a bad answer. The rubric is what stops the model producing | |
| one, and for several of these the difference is the product's credibility rather | |
| than a validation error: | |
| * **`cattle_skin` must never name a disease.** Lumpy skin disease and | |
| foot-and-mouth are notifiable under Nigeria's Animal Diseases (Control) Act | |
| 1988 s.8(1). A farm acting on an unqualified name from a photograph is a | |
| reporting obligation triggered by a guess. | |
| * **`cattle_age_dentition` gives a band and never a chronological age.** | |
| Teeth carry an eruption stage; a birth year is not in the photograph. | |
| * **`cattle_sex` must never block registration.** It is a suggestion beside a | |
| field the person fills in. | |
| * **`poultry_litter` may describe condition and never moisture.** Caking and | |
| friability are visible; water content is not, and it is the number a farm | |
| would act on. | |
| * **`egg_quality` reads external quality only.** Cracks need candling — RGB | |
| cannot see them — and cracks are the valuable half, so the limit is stated | |
| rather than left for a farm to discover. | |
| * **`poultry_footpad` reports a sampled prevalence, never a house figure.** | |
| ## What this module cannot do, and what has to happen instead | |
| Everything downstream of the answer checks that it is *well formed*: the schema | |
| is closed, the claim is in the vocabulary, the sentence is qualified, the number | |
| is in range. **Nothing here can catch an answer that is well formed and wrong.** | |
| A hosted model that confidently reads a healthy footpad as grade 3 produces a | |
| response that passes every gate in this repository. | |
| So the measurement in `experiments/` is not optional polish for these sixteen, | |
| it is the control. Until one exists, what stands between the model and a farm | |
| is: `experimental` on all sixteen, `requires_review` on every interpretation | |
| this runner builds, and `STANDING_WARNING` on every result. | |
| **`HUMAN_CONFIRMATION` is not among them, and an earlier draft of this file said | |
| it was.** Six of the sixteen carry it — `cattle_age_dentition`, `cattle_bcs`, | |
| `cattle_breed`, `cattle_sex`, `poultry_fecal`, `poultry_footpad`. **Ten do | |
| not**, and the ten include `cattle_skin` and both wound capabilities: | |
| cattle_feces cattle_hoof cattle_skin cattle_wound | |
| egg_quality poultry_eye_head poultry_feather poultry_hock | |
| poultry_litter poultry_wound | |
| The registry's reason is consistent: those ten declare an **observation** output | |
| — a description of the picture — and the directive requires confirmation where a | |
| capability makes a *judgement*. `requires_review` on every interpretation is that | |
| line drawn one level down, and it is the line this runner enforces. | |
| Whether an unmeasured hosted model's "description" of a lesion is an observation | |
| in that sense is a product question and not this module's to settle. It is | |
| written here because the alternative is a reader assuming, as this file once | |
| asserted, that all sixteen are behind a confirm button. | |
| """ | |
| from __future__ import annotations | |
| import logging | |
| from uuid import UUID, uuid4 | |
| from app.adapters import claims | |
| from app.adapters.base import AdapterError | |
| from app.adapters.claims import ContractViolation | |
| from app.adapters.multimodal import HostedMultimodalAdapter | |
| from app.adapters.transports import transport_from_env | |
| from app.capabilities import Capability | |
| from app.media import MediaRef, MediaStore | |
| from app.providers import ModelArtefact | |
| from app.quality import assess | |
| from app.schemas import ( | |
| InferenceLocation, | |
| InferenceResult, | |
| Interpretation, | |
| Observation, | |
| ) | |
| logger = logging.getLogger(__name__) | |
| #: What `Capability.model_provider` says for a capability answered over HTTP. | |
| #: | |
| #: A stable declaration rather than the model's name. The name of the model that | |
| #: actually ran is a fact about a *run* and travels on the result — see | |
| #: `providers.HostedModelProvider`, which reads it at call time so a rotation is | |
| #: a restart rather than a rebuild, and so the release row records which model | |
| #: produced which evidence. | |
| HOSTED_PROVIDER = "hosted-multimodal" | |
| #: The rubric for each capability, keyed by capability key. | |
| #: | |
| #: Read against each entry's `notes` in `app/capabilities.py`, which carry the | |
| #: directive section and the constraint. Where the two could disagree the | |
| #: registry wins and this is wrong. | |
| RUBRICS: dict[str, str] = { | |
| # -- cattle ----------------------------------------------------------- | |
| "cattle_bcs": ( | |
| "Score the body condition of the animal on the 1-5 scale, where 1 is " | |
| "emaciated and 5 is obese. Look at the hooks, the pins, the tail head " | |
| "and the ribs. Give a range covering the plausible scores, not a point " | |
| "value: a single view of one animal does not support a precise score. " | |
| "If the animal is rugged, muddy, heavily haired or standing so the " | |
| "hindquarter is hidden, say so in `limits` and widen the range." | |
| ), | |
| "cattle_age_dentition": ( | |
| "Count the permanent incisors that have erupted in the lower jaw and " | |
| "give the age BAND that eruption stage indicates. Report the count as " | |
| "an observation and the band as the claim.\n\n" | |
| "Teeth carry an eruption stage and not a birth year. Never state a " | |
| "chronological age, an age in months, or a year of birth — the band is " | |
| "the whole answer. If the mouth is not open, the lower incisors are not " | |
| "in frame, or wear makes the count ambiguous, set the count null and say " | |
| "which in `limits`.\n\n" | |
| "**This capability reads teeth and nothing else.** If the lower " | |
| "incisors are not visible in the photograph, refuse it — claim " | |
| "nothing and choose the limit that says so. Do not fall back on body " | |
| "size, frame, horn length or general appearance: those support a guess " | |
| "about age and this capability's whole claim is that its answer came " | |
| "from dentition. A band derived from the animal's build, published " | |
| "under `age_band`, is a different reading wearing this one's name." | |
| ), | |
| "cattle_breed": ( | |
| "Say which breed the animal's phenotype most resembles, from coat " | |
| "colour and pattern, horn shape, hump, dewlap, ear carriage and frame.\n\n" | |
| "**Not forcing a breed is a required answer and not a failure.** Most " | |
| "cattle on a Nigerian smallholding are crossbred, and a crossbred animal " | |
| "must be allowed to stay crossbred: where the animal does not sit " | |
| "cleanly in one breed, claim `crossbred_or_uncertain` rather than the " | |
| "nearest name. Where it does, phrase it as a resemblance — a " | |
| "'White-Fulani-like phenotype' — because a photograph supports a " | |
| "phenotype and not a pedigree." | |
| ), | |
| "cattle_sex": ( | |
| "Say which sex the animal's visible anatomy suggests — udder, scrotum, " | |
| "sheath, frame and horn shape.\n\n" | |
| "This is a suggestion beside a field a person fills in, and it must " | |
| "never read as settled. If the relevant anatomy is not visible, or the " | |
| "animal is young enough that the frame is uninformative, set the claim " | |
| "null and say so in `limits`. An unclear answer costs nothing here; a " | |
| "confident wrong one is written onto an animal's record." | |
| ), | |
| "cattle_wound": ( | |
| "Describe what is visible at the wound: its approximate size relative to " | |
| "the animal, whether the edges are open or closed, whether there is " | |
| "discharge, swelling, or fly strike, and how fresh it appears.\n\n" | |
| "**Triage only.** There is no public dataset of livestock wounds, so " | |
| "this is a description that helps somebody decide whether to look, and " | |
| "never a grade, a stage, a healing time or a cause. Never say what " | |
| "caused the wound. Where the wound looks deep, contaminated, near a " | |
| "joint or eye, or extensive, recommend veterinary review." | |
| ), | |
| "cattle_skin": ( | |
| "Describe the skin: whether there are nodules and roughly how many and " | |
| "how widely spread, whether there is hair loss, crusting, scaling, " | |
| "thickening or discharge, and which body regions are affected.\n\n" | |
| "**Claim what you can actually see, not the vaguest thing that fits.** " | |
| "If there are raised nodules, claim `nodular_lesions_visible` — that is " | |
| "the finding a farmer can act on. Keep `abnormal_skin_pattern` for skin " | |
| "that is clearly wrong in some other way, and do not add it beside the " | |
| "nodule claim as padding. Measured on 282 photographs, the nodule claim " | |
| "is right 98 times in 100 when it is made and the general one is right " | |
| "71; a specific claim is worth more to a farm and it is also the more " | |
| "accurate one.\n\n" | |
| "**Never name a disease.** Lumpy skin disease and foot-and-mouth are " | |
| "notifiable under Nigeria's Animal Diseases (Control) Act 1988 s.8(1), " | |
| "and a name from a photograph is a legal obligation triggered by a " | |
| "guess. Say what the skin looks like and let the vet name it." | |
| ), | |
| "cattle_hoof": ( | |
| "Describe the hoof: overgrowth of the toe or heel, uneven wear, " | |
| "separation at the white line, visible cracks, swelling above the " | |
| "coronet, and any discharge or smell of the interdigital space that is " | |
| "visible as wetness or matting.\n\n" | |
| "A hoof read through mud or dung is a photograph of mud. If the foot is " | |
| "not lifted, not washed, or the sole is not in frame, say so in `limits` " | |
| "rather than reading through it." | |
| ), | |
| "cattle_feces": ( | |
| "Describe the dropping: its consistency on a firm-to-liquid scale, its " | |
| "colour, and whether undigested fibre, mucus, blood or visible worm " | |
| "segments are present.\n\n" | |
| "Consistency reflects the current diet as much as health, so describe " | |
| "what is there and let a person read it against what the animal is " | |
| "eating. Never name a parasite species from a photograph of a dropping." | |
| ), | |
| # -- poultry ---------------------------------------------------------- | |
| "poultry_fecal": ( | |
| "Describe each dropping visible in the sample: its consistency, its " | |
| "colour, and whether blood, mucus, undigested feed or unusually watery " | |
| "content is present. Report how many droppings you could actually " | |
| "assess as an observation.\n\n" | |
| "This is a screen over a sample and never a house diagnosis. Describe " | |
| "the droppings and let a person decide; where blood is visible, " | |
| "recommend review rather than naming coccidiosis or any other disease." | |
| ), | |
| "poultry_footpad": ( | |
| "Grade the footpad on the 0-4 pododermatitis rubric: 0 is a clean " | |
| "unblemished pad, 4 is a large deep lesion with necrosis. Report the " | |
| "grade for each bird you can assess and how many birds you assessed.\n\n" | |
| "**A sampled prevalence, never a house figure.** The birds in these " | |
| "images are the sample; say how many they were and never extrapolate to " | |
| "the flock. If a pad is obscured by litter or the bird is not held so " | |
| "the pad faces the camera, exclude it and say so." | |
| ), | |
| "poultry_hock": ( | |
| "Describe the hock: whether there is discolouration, a burn, a scab, " | |
| "swelling or ulceration, and roughly how severe. Report how many birds " | |
| "you assessed.\n\n" | |
| "As with the footpad, this is a sample and not a house figure. Hock " | |
| "burn and footpad lesions often travel together but are separate " | |
| "captures and separate readings; do not infer one from the other." | |
| ), | |
| "poultry_feather": ( | |
| "Describe feather cover over the back and rump: whether coverage is " | |
| "complete, thinning or bare, whether there are broken shafts, and " | |
| "whether any bare skin shows pecking damage as distinct from moult.\n\n" | |
| "Report what the cover looks like. Do not attribute a cause — feather " | |
| "loss from pecking, moult, rubbing and nutrition look alike in one " | |
| "photograph, and naming a cause is what turns a description into a " | |
| "management decision nobody checked." | |
| ), | |
| "poultry_wound": ( | |
| "Describe what is visible at the wound: approximate size, whether the " | |
| "skin is broken, whether there is bleeding, scabbing or discharge, and " | |
| "whether the surrounding feathers are damaged.\n\n" | |
| "**Triage only**, as for cattle. Never state a cause and never a healing " | |
| "time. A bird with an open bleeding wound in a populated house needs " | |
| "separating, and saying so is the useful output." | |
| ), | |
| "poultry_eye_head": ( | |
| "Describe the head and eyes: whether the eyes are clear or show " | |
| "discharge, swelling or closure, whether there is facial or sinus " | |
| "swelling, whether the comb and wattles are of normal colour or are " | |
| "pale, blue or discoloured, and whether nasal discharge is visible.\n\n" | |
| "Comb colour and facial swelling are the two findings a farm acts on " | |
| "fastest and the two most often over-read from one photograph. Describe " | |
| "them plainly, name no disease, and recommend review where several " | |
| "findings appear together." | |
| ), | |
| "poultry_litter": ( | |
| "Describe the litter: whether the surface is loose and friable or caked " | |
| "and compacted, whether there is visible capping, and how much of the " | |
| "photographed area is affected.\n\n" | |
| "**Condition is visible. Moisture is not, and must never be claimed.** " | |
| "Caking correlates with moisture and is not a measurement of it, and " | |
| "moisture percentage is the number a farm would act on. Describe the " | |
| "surface; a farm measures water with a meter." | |
| ), | |
| "egg_quality": ( | |
| "Describe the external quality of the eggs visible: shell cleanliness, " | |
| "visible soiling, shape abnormality, ridging or roughness of the shell, " | |
| "and obvious breakage. Report how many eggs you could assess.\n\n" | |
| "**External quality only, and say so.** A crack that has not broken the " | |
| "shell membrane is invisible to a camera and needs candling, and cracks " | |
| "are the valuable half of egg quality — so a clean-looking tray here is " | |
| "not a tray without cracks. State that limit rather than leaving a farm " | |
| "to infer it." | |
| ), | |
| } | |
| #: Said on every run of every capability here, whatever the answer. | |
| #: | |
| #: Not a hedge for its own sake. These sixteen have no measurement — no | |
| #: `experiments/` run has put real photographs through any of them — and a | |
| #: hosted model's confident wrong answer is well-formed, so the thing that | |
| #: protects a farm is a person looking. `HUMAN_CONFIRMATION` makes that | |
| #: structural and this makes it visible. | |
| #: Appended to every rubric. The one instruction that applies to all sixteen. | |
| #: | |
| #: **Written because a measurement showed it was needed.** On `cattle_skin` the | |
| #: model reached for the vague claim more than twice as often as the specific | |
| #: one, and the specific one was the more accurate of the two — 97.9% against | |
| #: 71.4%. A model hedging toward the general claim is not being careful; it is | |
| #: producing a sentence a farm cannot act on, and being more wrong while it does. | |
| BE_SPECIFIC = ( | |
| "Prefer the most specific claim the picture actually supports. A precise " | |
| "finding is what a farmer can act on, and a vague one costs them a decision. " | |
| "Do not add a general claim beside a specific one as a hedge — if you can " | |
| "see the specific thing, claim only that. If you genuinely cannot tell, " | |
| "claim nothing and say why in `limits`; that is a better answer than a " | |
| "claim nobody can use." | |
| ) | |
| STANDING_WARNING = ( | |
| "Read by an experimental visual model, and not measured on livestock like " | |
| "these. Treat it as a second opinion on the photograph and confirm it." | |
| ) | |
| class ReasoningRunner: | |
| """Runs one hosted-multimodal capability against one capture. | |
| Constructed per capability key so `RUNNERS` maps a key to a runner exactly | |
| as `counting` and `identification` do, and so the rubric is bound at import | |
| rather than looked up on every request. | |
| """ | |
| def __init__(self, capability_key: str, adapter: HostedMultimodalAdapter | None = None): | |
| self.capability_key = capability_key | |
| self.rubric = RUBRICS[capability_key] | |
| # Injectable for the tests, which supply an adapter with a stub | |
| # transport. `None` means build one from the environment at call time, | |
| # which is what keeps a key rotation a restart. | |
| self._adapter = adapter | |
| def _vocabulary_note(self, capability: Capability) -> str: | |
| """The words and array bounds the schema knows and cannot send. | |
| ## Why this exists at all | |
| Two controls in `claims.schema_for` do not survive the trip to a | |
| provider, and the model then breaks them in the obvious way: | |
| * **`ClaimQuantity.values`.** `check_observation_values` refuses an | |
| observation whose `value` is a word the claim did not declare, and | |
| the schema has nowhere to put those words — `value` is | |
| `number | string | null`, and structured outputs refuses an `enum` | |
| beside a union type. So the model is told `body_condition_band` may | |
| carry a word and never which words, and the first real call answered | |
| `"Appropriate"` and then `"Thin to Appropriate"`. Both are refused, | |
| both are the model doing its best with what it was given, and both | |
| cost a vendor call. | |
| * **`maxItems`.** `_for_structured_output` strips it because the API | |
| refuses it, which leaves every array unbounded. The first real call | |
| also produced an answer truncated at the token ceiling — a runaway | |
| list — and a truncated answer is a whole capture refused. | |
| `claims.enforce` still catches both, so this is not a control. It is what | |
| stops the model failing a control it was never shown, which is the | |
| difference between a capability that works and one that retries. | |
| **Generated from the registry, never typed.** A word list written here | |
| would be a second copy of `ClaimQuantity.values` and would drift from | |
| the check that enforces it. | |
| """ | |
| contract = claims.schema_for(capability) | |
| output = capability.acquisition.output | |
| lines: list[str] = [] | |
| exact, named, wordless = [], [], [] | |
| for quantity in (output.quantities or ()): | |
| permitted, free_reason = output.words_for(quantity.claim) | |
| if free_reason is not None: | |
| named.append(quantity.claim) | |
| elif permitted: | |
| offered = ", ".join(f'"{w}"' for w in permitted) | |
| exact.append(f" - {quantity.claim}: {offered}") | |
| else: | |
| wordless.append(quantity.claim) | |
| if exact: | |
| lines.append( | |
| "Where an observation's `value` is a word it must be exactly " | |
| "one of the words listed for that claim — lower case, as " | |
| "written, not a phrase and not a pair joined by 'to'. If the " | |
| "picture sits between two of them choose the nearer one and " | |
| "widen the range instead.\n" + "\n".join(exact) | |
| ) | |
| if named: | |
| lines.append( | |
| f"These claims carry a NAME and nothing else: " | |
| f"{', '.join(named)}. A name is at most {claims._NAME_WORDS} " | |
| f"words and {claims._NAME_LENGTH} characters, with no digit and " | |
| f"no punctuation beyond a hyphen or an apostrophe — " | |
| f'"White Fulani", not a sentence describing one. Put the ' | |
| f"reasoning in `evidence`, which is a closed list, and never " | |
| f"here." | |
| ) | |
| if wordless: | |
| lines.append( | |
| f"These claims carry NO word: {', '.join(wordless)}. The claim " | |
| f"itself is the finding. Their `value` is a number or null, " | |
| f"never text." | |
| ) | |
| bounds = [] | |
| for field, spec in (contract.get("properties") or {}).items(): | |
| ceiling = spec.get("maxItems") if isinstance(spec, dict) else None | |
| if isinstance(ceiling, int): | |
| bounds.append(f"{field} at most {ceiling}") | |
| if bounds: | |
| lines.append( | |
| "Length limits, which the schema cannot express and which are " | |
| "checked after you answer: " + "; ".join(bounds) + "." | |
| ) | |
| if output.show_range: | |
| lines.append( | |
| "`range` is two DIFFERENT numbers, low then high. A pair like " | |
| "[0, 0] is an exact figure wearing a range's clothes and the " | |
| "whole answer is discarded for it.\n\n" | |
| "If what this capability measures is NOT VISIBLE in the " | |
| "picture, refuse: choose the limit that says the subject is not " | |
| "in frame, claim nothing, and let a person take it again. Do " | |
| "not answer with zeros, and do not answer with the widest band " | |
| "the scale allows either — a band from one end of the scale to " | |
| "the other is not a cautious reading, it is a refusal a farmer " | |
| "has to decode, and it reaches them looking like a measurement. " | |
| "Widen a band because the picture is poor; refuse because the " | |
| "subject is absent. They are different answers." | |
| ) | |
| return "\n\n".join(lines) | |
| def _reasoner(self) -> HostedMultimodalAdapter: | |
| if self._adapter is not None: | |
| return self._adapter | |
| return HostedMultimodalAdapter(transport=transport_from_env()).load() | |
| def run( | |
| self, | |
| *, | |
| request, | |
| capability: Capability, | |
| artefact: ModelArtefact, | |
| store: MediaStore, | |
| request_id: UUID | None = None, | |
| ) -> InferenceResult: | |
| request_id = request_id or uuid4() | |
| warnings: list[str] = [STANDING_WARNING] | |
| images = [] | |
| for media_id in request.media_ids: | |
| images.append(store.open_image(MediaRef( | |
| media_id=media_id, | |
| farm_id=request.farm_id, | |
| captured_at=request.captured_at, | |
| object_path=request.path_for(media_id), | |
| ))) | |
| # **The quality gate runs before the call, not after it.** A blurred or | |
| # dark frame is refused here for free; sending it costs a vendor call | |
| # and returns a confident reading of a photograph nobody could read. | |
| verdict = assess(images[0]) | |
| checks = list(verdict.checks) | |
| if verdict.blocked: | |
| return self._result( | |
| request=request, capability=capability, artefact=artefact, | |
| request_id=request_id, observations=[], interpretations=[], | |
| confidence=None, checks=checks, | |
| warnings=warnings + [ | |
| "The photograph was not clear enough to read, so nothing was " | |
| "sent to the model and nothing is claimed." | |
| ], | |
| recapture=True, | |
| ) | |
| try: | |
| note = self._vocabulary_note(capability) | |
| rubric = f"{self.rubric}\n\n{BE_SPECIFIC}" | |
| response = self._reasoner().reason( | |
| images, capability=capability, | |
| rubric=f"{rubric}\n\n{note}" if note else rubric, | |
| ) | |
| except ContractViolation as exc: | |
| # **Answered, and the answer may not be published.** Distinct from | |
| # the transport failure below, and the distinction decides whether | |
| # the run is retried. | |
| # | |
| # The model produced something; it broke the schema, the vocabulary, | |
| # a numeric bound or the forbidden-claims gate. Re-asking the same | |
| # photograph asks the same question of the same model and gets the | |
| # same answer, so raising here would queue a capture that fails | |
| # identically for ever — `MAX_ATTEMPTS` eventually stops it, at the | |
| # cost of that many vendor calls. | |
| # | |
| # The commonest cause is an out-of-protocol capture. `cattle_wound` | |
| # is a `lesion_close_up`; photograph a whole herd with it and the | |
| # honest answer is that there is no wound, which the contract has no | |
| # way to express — §7 requires a band from a capability that | |
| # declares one, and `[0, 0]` is an exact figure wearing a band's | |
| # clothes. Both rules are right and together they leave the model | |
| # nowhere to stand, so the capture is refused rather than the rules | |
| # bent. | |
| # | |
| # A farm gets a recapture prompt, nothing is claimed, and the reason | |
| # is on the run. | |
| logger.info( | |
| "%s could not be published for %s: %s", | |
| capability.key, request.media_ids[0], exc, | |
| ) | |
| return self._result( | |
| request=request, capability=capability, artefact=artefact, | |
| request_id=request_id, observations=[], interpretations=[], | |
| confidence=None, checks=checks, | |
| warnings=warnings + [ | |
| "This photograph did not support a reading this capability " | |
| "may publish, so nothing is claimed for it. Take it again " | |
| "the way the capture screen describes.", | |
| ], | |
| recapture=True, | |
| ) | |
| except AdapterError as exc: | |
| # **Not answered.** A timeout, a refused key, a 5xx, a truncated | |
| # body. `/jobs` turns a ValueError into a FAILED job which the API | |
| # keeps queued, and that is right here: the question was never put, | |
| # so asking it again is a different attempt rather than the same | |
| # one. | |
| raise ValueError(str(exc)) from exc | |
| warnings.extend(response.warnings) | |
| return self._from_envelope( | |
| parsed=response.parsed, request=request, capability=capability, | |
| artefact=artefact, request_id=request_id, checks=checks, | |
| warnings=warnings, | |
| ) | |
| def _from_envelope( | |
| self, *, parsed: dict, request, capability, artefact, request_id, | |
| checks, warnings, | |
| ) -> InferenceResult: | |
| """`claims.schema_for`'s envelope as an `InferenceResult`. | |
| Everything here has already been through `claims.parse_strict` and | |
| `claims.enforce`, so the vocabulary, the bounds and the forbidden-claim | |
| gate have all run. What is left is a transcription. | |
| """ | |
| observations = [ | |
| Observation( | |
| type=str(item["type"]), | |
| value=item.get("value"), | |
| unit=item.get("unit"), | |
| confidence=item.get("confidence"), | |
| ) | |
| for item in (parsed.get("observations") or []) | |
| if isinstance(item, dict) and item.get("type") | |
| ] | |
| # A ranged capability carries its answer in `range`, and the range is | |
| # the answer — `best_estimate` is emitted beside it and never instead of | |
| # it, because §7's whole complaint is a point value standing in for a | |
| # band. | |
| band = parsed.get("range") | |
| if isinstance(band, list) and len(band) == 2: | |
| output_type = capability.acquisition.output.type | |
| observations.append(Observation( | |
| type=output_type, value=float(band[0]), unit="range_low", | |
| confidence=None, | |
| )) | |
| observations.append(Observation( | |
| type=output_type, value=float(band[1]), unit="range_high", | |
| confidence=None, | |
| )) | |
| interpretations = [ | |
| Interpretation( | |
| label=str(item["label"]), | |
| confidence=item.get("confidence"), | |
| # Always true, and the schema pins it to `const: true`. A | |
| # hosted reading a person has not seen is not a record. | |
| requires_review=True, | |
| ) | |
| for item in (parsed.get("interpretations") or []) | |
| if isinstance(item, dict) and item.get("label") | |
| ] | |
| # The closed lists travel as warnings rather than as claims. They are | |
| # what the model was allowed to say *about its own answer*, and a farm | |
| # reading "One view only" beside a band is the point of them. | |
| for limit in (parsed.get("limits") or []): | |
| if isinstance(limit, str) and limit.strip(): | |
| warnings.append(limit.strip()) | |
| return self._result( | |
| request=request, capability=capability, artefact=artefact, | |
| request_id=request_id, observations=observations, | |
| interpretations=interpretations, | |
| confidence=parsed.get("confidence"), | |
| checks=checks, warnings=warnings, recapture=False, | |
| ) | |
| def _result( | |
| self, *, request, capability, artefact, request_id, observations, | |
| interpretations, confidence, checks, warnings, recapture, | |
| ) -> InferenceResult: | |
| banned = set(capability.acquisition.forbidden_claims) | |
| offending = sorted( | |
| {o.type for o in observations if o.type in banned} | |
| | {i.label for i in interpretations if i.label in banned} | |
| ) | |
| if offending: | |
| # `claims.enforce` already refuses these, and this refuses them | |
| # again on the way out. The gate that matters is the one closest to | |
| # the thing being protected, and a runner is the last place a | |
| # forbidden claim could be introduced. | |
| raise ValueError( | |
| f"{capability.key} tried to emit {offending}, which its own " | |
| f"registry entry forbids by name." | |
| ) | |
| return InferenceResult( | |
| request_id=request_id, | |
| capability_key=capability.key, | |
| model_id=artefact.model_id, | |
| model_version=artefact.version, | |
| inference_location=InferenceLocation.REMOTE, | |
| subject_type=request.subject_type, | |
| subject_id=request.subject_id, | |
| observations=observations, | |
| interpretations=interpretations, | |
| observation_confidence=confidence, | |
| # **Never set, on any path.** The three-level label beside an | |
| # interpretation would read as a confidence in the interpretation, | |
| # and nothing has calibrated one — these sixteen have no measurement | |
| # at all. `requires_review` is the honest field and it is always | |
| # true. | |
| interpretation_confidence=None, | |
| quality_checks=checks, | |
| warnings=warnings, | |
| recommended_recapture=recapture, | |
| ) | |
| #: Every capability a hosted multimodal model answers. | |
| #: | |
| #: Derived from `RUBRICS` rather than listed twice: a capability with a rubric | |
| #: and no runner would be silently unavailable, and a runner with no rubric | |
| #: cannot be constructed. | |
| RUNNERS: dict[str, ReasoningRunner] = { | |
| key: ReasoningRunner(key) for key in RUBRICS | |
| } | |