orbit-vision / README.md
paymansuper's picture
Update README.md
2e2e216 verified
|
Raw History Blame Contribute Delete
4.81 kB
metadata
license: cc-by-nc-4.0
language:
  - en
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - vision-language
  - vlm
  - orbit
  - starlex
  - siglip2
  - muon
  - research-preview
base_model:
  - StarlexAI/Orbit-Nano-Compliance-E2B
  - google/siglip2-so400m-patch14-384
extra_gated_heading: Orbit-Vision research preview
extra_gated_prompt: >-
  Orbit-Vision is a research preview for non-commercial research. Parts of its
  training data come from FineVision subsets whose licenses are still being
  reviewed. By requesting access you agree to use it for non-commercial research
  only.
extra_gated_fields:
  Affiliation: text
  Intended use: text
  I agree to use this model for non-commercial research only: checkbox

Orbit-Vision v1 (research preview)

Our own language model learned to read images in a single night. 60 % on TextVQA, and only 6.5 % when the image is blank, so it really looks.

orbit-vision_16x9_v2

Orbit-Vision answers questions about photos, charts and documents. Its language model is Orbit-Nano, pretrained from scratch by Starlex AI. The eyes are Google's SigLIP2-SO400M image encoder, kept frozen. Starlex AI trained the projector that connects the two and fine-tuned Orbit-Nano to use it.

Training about 11 hours on 4 GPUs, 94,356 steps
Language model Orbit-Nano, 4.03B parameters, 1.75B compute per token, pretrained by Starlex AI
Image encoder SigLIP2-SO400M by Google, 384 px, frozen
Image tokens 81 per image
License CC-BY-NC-4.0, research preview

Results

Held-out FineVision VQA files, never used in training, 200 questions each, greedy answers, relaxed match. The blank image column asks the same questions with an empty image. The gap shows how much of each answer comes from actually reading the image.

Benchmark With image Blank image Gain from the image
TextVQA (text in photos) 60.0 % 6.5 % +53.5 points
ChartQA (charts) 41.5 % 9.5 % +32.0 points
DocVQA (documents) 35.0 % 3.0 % +32.0 points

Try it

pip install torch transformers safetensors huggingface_hub pillow
python orbit_vision.py photo.jpg "What is written on the sign?"
import torch
from orbit_vision import OrbitVision
m = OrbitVision(".", device="cuda", dtype=torch.bfloat16)
print(m.ask("chart.png", "Which year has the highest value?"))

SigLIP2 is downloaded from google/siglip2-so400m-patch14-384 on first use.

How it was built

  1. Projector. A 3 x 3 pixel shuffle turns SigLIP2's 729 patches into 81 tokens, then LayerNorm and a two layer MLP map them into Orbit-Nano's embedding space. The image tokens replace 81 placeholder tokens right after the start of the first user turn.
  2. Warm up. For the first 6 % of the run only the projector learns.
  3. Joint training. Then projector and Orbit-Nano learn together, Muon on Orbit's matrices and AdamW on embeddings, head and projector, WSD schedule, sequence length 1,024.
  4. Data. FineVision subsets densefusion_1m, allava_laion, DoclingMatix, lnqa, docvqa, chartqa and textvqa (single image samples), plus 12 % text only chat (SmolTalk) against forgetting.

Limitations

  • Research preview. Single image at 384 x 384 px with 81 image tokens, so small text in documents is often misread.
  • English only. Orbit-Nano's chat ability got weaker during vision training (text validation loss 0.73 to 0.96). For text only use, take Orbit-Nano itself.
  • Can describe things that are not in the image. No safety tuning beyond the base model.

The Orbit family

Model What it does
Orbit-Nano-Compliance-E2B German and English compliance chat, rank 1 of 7 on our compliance benchmark
Orbit-Nano-Compliance-E2B-GGUF The same model for laptops
Orbit Phone Speech recognition, 2.60 % WER on LibriSpeech test-clean
Orbit-Vision Questions about images, charts and documents
Orbit-Image Pixel art from text (research preview)

License

CC-BY-NC-4.0 for the projector and the fine-tuned language model weights in this repository, until the training data licenses are fully reviewed. SigLIP2 keeps its Apache-2.0 license.

Built by Starlex AI, a Foundation AI Lab.