| --- |
| license: cc-by-nc-sa-4.0 |
| base_model: |
| - convaiinnovations/laya |
| - google/siglip-base-patch16-224 |
| tags: |
| - laya |
| - vision |
| - image-classification |
| - visual-question-answering |
| - decision-making |
| - calibration |
| --- |
| |
| # Laya-Vision |
|
|
| Image in, calibrated typed decision out. Laya-Vision gives [Laya](https://huggingface.co/convaiinnovations/laya) image states. A frozen SigLIP-base/16 tower is pooled to 49 tokens, which are projected into Laya's embedding space. The model then answers `choice`, `noul` (yes/no) and `score` questions about the image with calibrated probabilities. Text-only requests take Laya's normal text path. |
|
|
| ## Usage |
|
|
| Requires the `laya_vision` package from the project repo (`pip install -e .`). |
|
|
| ```python |
| import laya_vision |
| |
| agent = laya_vision.load("ZeraG07/laya-vision", device="cuda") # or "cpu"; private repo: set HF_TOKEN |
| out = agent.predict( |
| {"image": "photo.jpg"}, # path, bytes, base64 or PIL image; optional "text": "..." |
| { |
| "land": {"type": "choice", "instructions": "What land cover is shown?", |
| "criteria": ["river", "forest", "highway", "farmland"]}, |
| "water": {"type": "noul", "instructions": "Is there water in this image?"}, |
| "green": {"type": "score", "instructions": "How much vegetation is visible?", |
| "criteria": ["none", "some", "a lot"]}, |
| }, |
| ) |
| out["answers"]["water"]["noul"] # calibrated P(yes) |
| ``` |
|
|
| HTTP server, using Laya's `/v1/systemone` contract with base64 images: |
|
|
| ```bash |
| python -m laya_vision.serve --checkpoint ZeraG07/laya-vision --port 8000 --device cuda |
| ``` |
|
|
| ## Results |
|
|
| Held-out test splits. The acceptance criteria come from the project's `TRAINING.md`. **All four checks pass.** |
|
|
| | Task | Laya-Vision | caption→Laya (BLIP caption + stock Laya) | stock Laya | |
| |---|---|---|---| |
| | EuroSAT (choice) | **0.970** | 0.389 | | |
| | Oxford-IIIT Pets (choice + "is this a …?") | **0.765** | 0.280 | | |
| | VQAv2 yes/no | **0.608** | 0.592 | | |
| | BoolQ (text) | 0.747 | | 0.756 | |
| | AG News (text) | 0.924 | | 0.926 | |
|
|
| - **Image calibration:** ECE@10 is 0.014. |
| - **Latency:** p50 GPU latency for one question is 1.68× text-only Laya (RTX 4060 Ti). |
|
|
| ## Training |
|
|
| The recipe is "B" in `TRAINING.md`. Everything was trained on one 8 GB RTX 4060 Ti. |
|
|
| **Stage 1 (alignment).** COCO caption matching with easy negatives (444k rows), plus 10% typed-decision text rows. |
| - Trained: the projector at LR 1e-4, and LoRA (r=16) on Laya's encoder. |
| - Projector guards: an input LayerNorm, running standardization of the output, and a frozen modality embedding. They prevent the modality collapse seen with a projector-only stage 1 at LR 1e-3. |
|
|
| **Stage 2 (decisions).** Ran for 2 epochs with LoRA, from stage 1. |
| - Image tasks: CIFAR-10, Oxford Pets, Food-101, EuroSAT, A-OKVQA, ScienceQA (image subset) and VQAv2 yes/no. |
| - Text: 30% typed-decision rows. |
|
|
| **Stage 2c.** A continuation that adds 160k VQAv2 yes/no questions from the train split. |
|
|
| **WiSE-FT.** Laya's weights are interpolated toward stock Laya: `stock + 0.85 · (fine-tuned − stock)`. |
| - This limits text regression. |
| - α was chosen on a BoolQ train-split slice and on the image calib splits, never on the test sets. |
|
|
| **Calibration.** Per-modality temperature scaling on the calib splits. |
|
|
| Inside the checkpoint: |
| - `vision.safetensors` holds the pooler and projector. The SigLIP tower is frozen and downloaded from the Hub. |
| - `model.safetensors` is a plain Laya state dict. |
| - `rl_agent_config.json["vision"]` records the tower, the projector flags, `wise_alpha` and the image temperatures. |
|
|
| ## Limitations |
|
|
| - **Resolution.** Images are seen at 224×224 as 49 pooled tokens, so fine detail is limited. VQAv2 is only slightly above the caption baseline. |
| - **Untested tasks.** KonIQ (quality score) was not trained or evaluated. `score` questions about images work mechanically but are uncalibrated for image tasks. |
| - **Possible image overlap.** The stage-1 COCO captions (Karpathy split) include some val2014 images, and the VQAv2 test questions also come from val2014. Answers never overlapped. |
|
|
| ## License |
|
|
| This model is released under **CC BY-NC-SA 4.0** (non-commercial), the most restrictive licence among its training data (ScienceQA). |
| - Laya and SigLIP are Apache-2.0. |
| - Several training sets have unclear or research-only image licences (COCO/Flickr, CIFAR-10, Food-101). |
| - See `laya_vision/data/LICENSES.md` in the project repo. |
|
|
