orbit-vision / README.md
paymansuper's picture
Update README.md
2e2e216 verified
|
Raw History Blame Contribute Delete
4.81 kB
---
license: cc-by-nc-4.0
language: [en]
library_name: transformers
pipeline_tag: image-text-to-text
tags: [vision-language, vlm, orbit, starlex, siglip2, muon, research-preview]
base_model: [StarlexAI/Orbit-Nano-Compliance-E2B, google/siglip2-so400m-patch14-384]
extra_gated_heading: Orbit-Vision research preview
extra_gated_prompt: >-
Orbit-Vision is a research preview for non-commercial research. Parts of its training data come from FineVision
subsets whose licenses are still being reviewed. By requesting access you agree to use it for non-commercial
research only.
extra_gated_fields:
Affiliation: text
Intended use: text
I agree to use this model for non-commercial research only: checkbox
---
# Orbit-Vision v1 (research preview)
**Our own language model learned to read images in a single night. 60 % on TextVQA, and only 6.5 % when the image
is blank, so it really looks.**
![orbit-vision_16x9_v2](https://cdn-uploads.huggingface.co/production/uploads/6996621a93e0b3b5807ba110/TwQ23m56HruiOcZco8F5c.png)
Orbit-Vision answers questions about photos, charts and documents. Its language model is
[Orbit-Nano](https://huggingface.co/StarlexAI/Orbit-Nano-Compliance-E2B), pretrained from scratch by Starlex AI. The
eyes are Google's SigLIP2-SO400M image encoder, kept frozen. Starlex AI trained the projector that connects the two
and fine-tuned Orbit-Nano to use it.
| | |
|---|---|
| Training | about 11 hours on 4 GPUs, 94,356 steps |
| Language model | Orbit-Nano, 4.03B parameters, 1.75B compute per token, pretrained by Starlex AI |
| Image encoder | SigLIP2-SO400M by Google, 384 px, frozen |
| Image tokens | 81 per image |
| License | CC-BY-NC-4.0, research preview |
## Results
Held-out FineVision VQA files, never used in training, 200 questions each, greedy answers, relaxed match. The blank
image column asks the same questions with an empty image. The gap shows how much of each answer comes from actually
reading the image.
| Benchmark | With image | Blank image | Gain from the image |
|---|---|---|---|
| TextVQA (text in photos) | **60.0 %** | 6.5 % | +53.5 points |
| ChartQA (charts) | **41.5 %** | 9.5 % | +32.0 points |
| DocVQA (documents) | **35.0 %** | 3.0 % | +32.0 points |
## Try it
```bash
pip install torch transformers safetensors huggingface_hub pillow
python orbit_vision.py photo.jpg "What is written on the sign?"
```
```python
import torch
from orbit_vision import OrbitVision
m = OrbitVision(".", device="cuda", dtype=torch.bfloat16)
print(m.ask("chart.png", "Which year has the highest value?"))
```
SigLIP2 is downloaded from [google/siglip2-so400m-patch14-384](https://huggingface.co/google/siglip2-so400m-patch14-384)
on first use.
## How it was built
1. **Projector.** A 3 x 3 pixel shuffle turns SigLIP2's 729 patches into 81 tokens, then LayerNorm and a two layer
MLP map them into Orbit-Nano's embedding space. The image tokens replace 81 placeholder tokens right after the
start of the first user turn.
2. **Warm up.** For the first 6 % of the run only the projector learns.
3. **Joint training.** Then projector and Orbit-Nano learn together, Muon on Orbit's matrices and AdamW on
embeddings, head and projector, WSD schedule, sequence length 1,024.
4. **Data.** FineVision subsets densefusion_1m, allava_laion, DoclingMatix, lnqa, docvqa, chartqa and textvqa
(single image samples), plus 12 % text only chat (SmolTalk) against forgetting.
## Limitations
- Research preview. Single image at 384 x 384 px with 81 image tokens, so small text in documents is often misread.
- English only. Orbit-Nano's chat ability got weaker during vision training (text validation loss 0.73 to 0.96).
For text only use, take [Orbit-Nano](https://huggingface.co/StarlexAI/Orbit-Nano-Compliance-E2B) itself.
- Can describe things that are not in the image. No safety tuning beyond the base model.
## The Orbit family
| Model | What it does |
|---|---|
| [Orbit-Nano-Compliance-E2B](https://huggingface.co/StarlexAI/Orbit-Nano-Compliance-E2B) | German and English compliance chat, rank 1 of 7 on our compliance benchmark |
| [Orbit-Nano-Compliance-E2B-GGUF](https://huggingface.co/StarlexAI/Orbit-Nano-Compliance-E2B-GGUF) | The same model for laptops |
| [Orbit Phone](https://huggingface.co/StarlexAI/orbit-phone) | Speech recognition, 2.60 % WER on LibriSpeech test-clean |
| [Orbit-Vision](https://huggingface.co/StarlexAI/orbit-vision) | Questions about images, charts and documents |
| [Orbit-Image](https://huggingface.co/StarlexAI/orbit-image) | Pixel art from text (research preview) |
## License
CC-BY-NC-4.0 for the projector and the fine-tuned language model weights in this repository, until the training data
licenses are fully reviewed. SigLIP2 keeps its Apache-2.0 license.
Built by Starlex AI, a Foundation AI Lab.