deVision v0.2 card (github 458d9974ff8771520d2298926520dc235a398ed2)

MANIFEST.json

@@ -8,8 +8,8 @@
   "sha256": "cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30"
  },
  "README.md": {
-  "size": 13520,
-  "sha256": "584f165e73880275bb92348107839e7483a012ef3fa23ea7a6717576e97c05c4"
+  "size": 9556,
+  "sha256": "44f335c6cc7e9fa1cf33d6f24aed4269f78dd114e8cf0abfb9598b636f2b9462"
  },
  "config.json": {
   "size": 353,
@@ -33,7 +33,7 @@
  },
  "provenance.json": {
   "size": 1324,
-  "sha256": "94d064129af62edafbcf05396b0a0b589034218f829070967039e8a3126e152c"
+  "sha256": "99cd0254a1bc35a09b46beef7280ce5243379638eba0125d9a039c222efe654c"
  },
  "tokenizer/tokenizer.json": {
   "size": 3583228,

README.md

@@ -145,125 +145,46 @@ model-index:
 
 # deVision v0.2
 
-**Image + typed questions → structured answers and calibrated probabilities.** deVision is a non-autoregressive visual decision model built from Laya's ModernBERT-large decision model and a SigLIP2 vision encoder. It scores the supplied options without generating text. Multiple questions about one image share a single image encoding.
+**Image + English questions → structured answers with calibrated probabilities.** deVision pairs a SigLIP2 vision encoder with Laya's ModernBERT decision model. It scores the options of yes/no (`noul`) and multiple-choice (`choice`) questions instead of generating text, follows the Jev answer format, and runs without a GPU. Several questions about one image share a single image encoding.
 
-This is release **v0.2**. It supports English, one image per request, yes/no decisions (`noul`) and multiple-choice decisions (`choice`). Weights and code are released under the Apache-2.0 licence; see [Licence and data](#licence-and-data).
-
-## Install
+## Usage
 
 ```bash
 pip install "devision @ git+https://github.com/byebyebruce/devision"
 ```
 
-Python 3.11 or newer is required; the install includes the HTTP server and browser demo. If the model repository is private, authenticate first with `hf auth login` using an account that has access.
-
-## Python quickstart
-
 ```python
 import devision
 
-model = devision.load("lukbit/devision", revision="v0.2", device="cpu")
+model = devision.load("lukbit/devision")   # latest release; revision="v0.2" pins this one; device defaults to auto
 result = model.predict(
-    image="photo.jpg",  # an image path, an http(s) URL, base64, bytes or a PIL image
-    state="",           # text context; empty when there is none
+    image="photo.jpg",   # a path, an http(s) URL, a data URI, base64, bytes or a PIL image
+    state="",            # text context; "" when there is none
     questions={
         "has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"},
-        "room": {
-            "type": "choice",
-            "instructions": "Which room is this?",
-            "criteria": {"kitchen": None, "bathroom": None, "bedroom": None},
-        },
+        "room": {"type": "choice", "instructions": "Which room is this?",
+                 "criteria": {"kitchen": None, "bathroom": None, "bedroom": None}},
     },
 )
-print(result["answers"]["has_fork"]["noul"])       # probability of yes
-print(result["answers"]["room"]["choice"])         # selected option
-print(result["answers"]["room"]["probabilities"])  # probability of each option
+print(result["answers"]["has_fork"]["noul"])        # probability of yes
+print(result["answers"]["room"]["probabilities"])   # one probability per option, summing to 1
 ```
 
-`model.decide(...)` takes the same arguments. Pin `revision="v0.2"` so that later releases do not change your results. The default device is `auto` (CUDA, then MPS, then CPU).
-
-### Image input
-
-The Python `image=` argument accepts:
-
-| Input | Example |
-|---|---|
-| HTTP(S) URL | `image="https://example.com/photo.jpg"` |
-| Local path | `image="photo.jpg"` or `image=Path("photo.jpg")` |
-| Base64 data URI | `image="data:image/png;base64,..."` |
-| Plain base64 | `image=base64_string` |
-| Encoded image bytes | `image=image_bytes` (PNG / JPEG file contents) |
-| PIL image | `image=pil_image` |
-
-### Context
-
-`state` carries text context as a string, object or array, exactly as in Jev:
-
-```python
-result = model.decide(
-    image="photo.jpg",
-    state={"note": "The customer says this item arrived damaged."},
-    questions={"damaged": {"type": "noul", "instructions": "Is the item visibly damaged?"}},
-)
-```
-
-An `image` key inside an object state stays text: `state={"image": "photo.jpg"}` does not load the file. The Jev-compatible image part `state=[{"type": "image", "url": "https://..."}]` also works; do not combine it with `image=`.
-
-### Output
-
-- `noul`: the probability that the statement holds, between 0 and 1.
-- `choice`: the highest-probability option; `probabilities` covers every option and sums to 1.
-- `confidence` (choice) follows Jev: `(n * p_max - 1) / (n - 1)`.
-- Invalid requests raise `devision.InvalidRequest`.
-
-## HTTP API and browser demo
+HTTP server with a browser demo at `http://127.0.0.1:8000`:
 
 ```bash
-pip install "devision @ git+https://github.com/byebyebruce/devision"
-devision-serve --checkpoint lukbit/devision --device cpu --port 8000
+devision-serve --checkpoint lukbit/devision --port 8000
+curl -s http://127.0.0.1:8000/v1/systemone -H 'Content-Type: application/json' \
+  -d '{"image": "https://example.com/photo.jpg", "state": "", "questions": {"has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"}}}'
 ```
 
-Open `http://127.0.0.1:8000` for the demo. The server speaks the Jev-compatible `POST /v1/systemone` format; an image is an element of the `state` array:
-
-```json
-{"state": [{"type": "image", "url": "https://example.com/photo.jpg"}, "optional text"],
- "questions": {"has_fork": {"type": "noul", "instructions": "Is there a fork in the image?"}}}
-```
-
-Use `{"type": "image", "base64": "..."}` for an uploaded picture. HTTP never reads server-local paths. Invalid requests and `score` questions return HTTP 422.
-
-## Architecture
-
-| Component | |
-|---|---|
-| Vision encoder | Frozen SigLIP2-B/16, 256 × 256 input, about 93M parameters |
-| Image preprocessing | Aspect-preserving resize and letterbox padding |
-| Connector | 2 × 2 patch grouping, then LayerNorm → Linear → GELU → Linear |
-| Visual sequence | 64 visual tokens, inserted after `[CLS]` |
-| Text encoder | Laya-initialised ModernBERT-large, about 395M parameters |
-| Decision head | Laya's two Transformer layers and option scorer, about 27M parameters |
-| Checkpoint | About 518M parameters, FP32, 2.07 GB of weights (LoRA merged) |
-
-Each option is scored at its `[MASK]` position; a temperature-scaled softmax turns the scores into probabilities.
-
-## Training and calibration
+Over HTTP, `image` is an http(s) URL, a data URI or base64; the server never reads its own files. Responses follow Jev; `confidence` for `choice` is `(n * p_max - 1) / (n - 1)`. Invalid requests return 422 (`devision.InvalidRequest` in Python).
 
-deVision is trained in two stages. Stage 1 aligns the projector on COCO captions (image-conditioned masked words). Stage 2 trains the decision on yes / no and multiple-choice questions with Laya's RLCD objective (projector, decision head and a LoRA on ModernBERT; the LoRA is merged for release). Earlier training covered COCO-derived existence / position / size questions, VQAv2, GQA, A-OKVQA, ScienceQA, AI2D, TQA, VSR, Visual7W telling, mirrored left/right pairs and counting. The last training step continued from that checkpoint for one pass over 353,830 questions: an extension pack of 315,342 questions from thirteen public sources (Objects365, TallyQA, CLEVR, CLEVR-Math, Super-CLEVR, FigureQA, MapQA, IconQA, SNLI-VE, Vision-Flan, VisOnlyQA, SpatialSense, PixMo-Count) plus 38,488 replayed questions of the earlier abilities; 44,229 steps on Apple MPS in FP32, learning rates `5e-5` (projector, head) and `1e-4` (LoRA), 500 warm-up steps. Full configurations and results: the [training log](https://github.com/byebyebruce/devision/blob/master/docs/training-log.md).
-
-Temperatures fitted on the project's calibration set (a question's type / option-count bucket first, else its type):
-
-| Bucket | Temperature |
-|---|---:|
-| Choice, 2 options | 1.8836 |
-| Choice, 3–5 options | 1.9790 |
-| Noul (yes / no) | 1.6382 |
-| Choice, any other count | 1.9568 |
-
-Calibration changes probabilities, not visual ability. Validate confidence thresholds on your own data.
+**Good for:** whether something is present and how many (ask counts as a choice), what kind of thing or scene, colours and materials, up/down and left/right in photos (as a choice). **Not for:** comparing two places on a diagram, chart or map; relative-position yes/no questions; small text; other languages or several images. Use the probabilities as thresholds and send uncertain cases to a person or a stronger model, after checking the thresholds on your own data.
 
 ## Evaluation
 
-Results of this checkpoint through `decide`, with the fitted temperatures; ECE uses 15 bins. *Mismatched* is the accuracy when every picture is swapped for an unrelated one (how much the answer depends on the picture). The project test sets use held-out pictures; they are project-specific subsets or generated questions, not official leaderboard scores, and several have informed decisions across training rounds.
+Accuracy through `decide` with the fitted temperatures. *Mismatched*: the same questions with every picture swapped for an unrelated one. Test sets use held-out pictures; they are project subsets, not official leaderboard scores.
 
 | Set | Questions | Accuracy | Mismatched | ECE |
 |---|---:|---:|---:|---:|
@@ -279,9 +200,7 @@ Results of this checkpoint through `decide`, with the fitted temperatures; ECE u
 | Visual7W, project held-out split | 1,000 | 0.721 | 0.422 | 0.030 |
 | Fresh counting test (unseen pictures) | 600 | 0.740 | 0.507 | 0.065 |
 
-### Comparison with Laya Vision 201M
-
-Laya Vision's published per-question predictions on the same questions; differences in percentage points with 95% intervals from paired resampling by picture. We did not rerun Laya Vision.
+Against Laya Vision 201M on the same questions (its published per-question predictions; difference in points, 95% interval from paired resampling by picture):
 
 | Set | Questions | Laya Vision | deVision | Difference |
 |---|---:|---:|---:|---|
@@ -292,23 +211,30 @@ Laya Vision's published per-question predictions on the same questions; differen
 
 On the full POPE random / popular / adversarial sets (3,000 questions each) deVision scores **0.891 / 0.868 / 0.791**, against Laya Vision's published **0.836 / 0.819 / 0.777** (aggregate scores only, not paired).
 
-CPU latency: P50 148 ms, P95 161 ms (Darwin arm64, 4 threads, FP32, one question per request, warm-up excluded). Several questions about one picture in one request share the image encoding, so each extra question costs less.
+CPU latency: P50 148 ms, P95 161 ms (Darwin arm64, 4 threads, FP32, one question per request, warm-up excluded).
+
+## Model
+
+| | |
+|---|---|
+| Vision | Frozen SigLIP2-B/16, 256 × 256 letterboxed input, 64 visual tokens after a 2 × 2 merge and an MLP projector |
+| Decision | Laya-initialised ModernBERT-large and Laya's decision head; each option is scored at its `[MASK]`, then a temperature-scaled softmax |
+| Size | About 518M parameters, FP32, 2.07 GB (LoRA merged) |
+
+Training: the projector is first aligned on COCO captions, then the decision is trained with Laya's RLCD objective on yes/no and multiple-choice questions (projector, decision head and a LoRA on ModernBERT), round after round. This release continued for one pass over 353,830 questions, 315,342 of them from thirteen public datasets (Objects365, TallyQA, CLEVR, CLEVR-Math, Super-CLEVR, FigureQA, MapQA, IconQA, SNLI-VE, Vision-Flan, VisOnlyQA, SpatialSense, PixMo-Count) and the rest replaying earlier data (VQAv2, GQA, A-OKVQA, ScienceQA, VSR, Visual7W, COCO-derived questions). Temperatures are fitted per question type and option count. Details: the [training log](https://github.com/byebyebruce/devision/blob/master/docs/training-log.md).
 
 ## Limitations
 
-- **Scientific diagrams and charts remain difficult.** On the 323 ScienceQA natural-science questions that need the picture, this release is about 24 points behind Laya Vision (see the comparison table). Comparing two named regions of a picture (two magnet poles, two series of a chart, two regions of a map, two line segments) is the main open weakness; training on tens of thousands of such questions left it near chance.
-- **Spatial reasoning is incomplete.** A left/right question is answered right on both a picture and its mirror for about 72% of position pairs and 65% of relative-position choice pairs, but only 19% of relative-position yes / no pairs.
-- **Object presence leans towards "yes".** Compared with the previous checkpoint, it more often says an object is present when the annotation says it is not (COCO existence test 95.5% → 93.8%; POPE adversarial 79.1%).
-- **Small text and fine detail** are limited by the 256 × 256 input; this is not an OCR model.
-- **English, one image, `noul` and `choice` only.** Other languages, several images and `score` questions are not supported.
-- **Probabilities can be miscalibrated out of domain.** A low ECE on these benchmarks does not guarantee reliable confidence on your data.
+- **Comparing two places named in the question** (two magnet poles, two chart series, two map regions) stays near chance; on the ScienceQA questions that need the picture it is about 24 points behind Laya Vision.
+- **Relative-position yes/no questions** are weak: right on both a picture and its mirror for only 19% of pairs (65% for relative-position choice questions).
+- **Presence leans towards "yes"**: it says an absent object is there more often than the previous release (POPE adversarial 0.791).
+- English only, one image, `noul` and `choice` only; 256 × 256 input, so no small text.
+- Calibration was fitted on this project's data and may not hold on yours.
 
-## Licence and data
+## Licence
 
-The weights and the code are released under the **Apache-2.0** licence, the licence of the three base models ([Laya](https://huggingface.co/convaiinnovations/laya), [SigLIP2](https://huggingface.co/google/siglip2-base-patch16-256), [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-large)). The training data come from public datasets with their own terms, some of them non-commercial (for example ScienceQA, CC BY-NC-SA 4.0) or covering the pictures separately (COCO / Flickr images). Whether such terms carry over to trained weights is not settled; check the datasets listed in this card's metadata against your use.
+Weights and code: Apache-2.0, like the three base models ([Laya](https://huggingface.co/convaiinnovations/laya), [SigLIP2](https://huggingface.co/google/siglip2-base-patch16-256), [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-large)). The training datasets (listed in this card's metadata) have their own terms, some non-commercial (e.g. ScienceQA, CC BY-NC-SA 4.0); check them against your use.
 
 ## Links
 
-- [Code, training configurations and experiment records](https://github.com/byebyebruce/devision) — this release is tag `model-v0.2`
-- [Evaluation details](evaluation/results.md) · [metrics](evaluation/results.json) · [provenance](provenance.json)
-- [Laya decision model](https://huggingface.co/convaiinnovations/laya) · [SigLIP2](https://huggingface.co/google/siglip2-base-patch16-256) · [Laya Vision](https://github.com/r33drichards/laya-vision)
+[Code and training records](https://github.com/byebyebruce/devision) (this release: tag `model-v0.2`) · [evaluation details](evaluation/results.md) · [provenance](provenance.json) · [Laya Vision](https://github.com/r33drichards/laya-vision)