How to use from the
Use from the
Transformers library
# Gated model: Login with a HF token with gated access permission
hf auth login
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="StarlexAI/orbit-vision", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("StarlexAI/orbit-vision", trust_remote_code=True, device_map="auto")
Quick Links

Orbit-Vision research preview

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Orbit-Vision is a research preview for non-commercial research. Parts of its training data come from FineVision subsets whose licenses are still being reviewed. By requesting access you agree to use it for non-commercial research only.

Log in or Sign Up to review the conditions and access this model content.

Orbit-Vision v1 (research preview)

Our own language model learned to read images in a single night. 60 % on TextVQA, and only 6.5 % when the image is blank, so it really looks.

orbit-vision_16x9_v2

Orbit-Vision answers questions about photos, charts and documents. Its language model is Orbit-Nano, pretrained from scratch by Starlex AI. The eyes are Google's SigLIP2-SO400M image encoder, kept frozen. Starlex AI trained the projector that connects the two and fine-tuned Orbit-Nano to use it.

Training about 11 hours on 4 GPUs, 94,356 steps
Language model Orbit-Nano, 4.03B parameters, 1.75B compute per token, pretrained by Starlex AI
Image encoder SigLIP2-SO400M by Google, 384 px, frozen
Image tokens 81 per image
License CC-BY-NC-4.0, research preview

Results

Held-out FineVision VQA files, never used in training, 200 questions each, greedy answers, relaxed match. The blank image column asks the same questions with an empty image. The gap shows how much of each answer comes from actually reading the image.

Benchmark With image Blank image Gain from the image
TextVQA (text in photos) 60.0 % 6.5 % +53.5 points
ChartQA (charts) 41.5 % 9.5 % +32.0 points
DocVQA (documents) 35.0 % 3.0 % +32.0 points

Try it

pip install torch transformers safetensors huggingface_hub pillow
python orbit_vision.py photo.jpg "What is written on the sign?"
import torch
from orbit_vision import OrbitVision
m = OrbitVision(".", device="cuda", dtype=torch.bfloat16)
print(m.ask("chart.png", "Which year has the highest value?"))

SigLIP2 is downloaded from google/siglip2-so400m-patch14-384 on first use.

How it was built

  1. Projector. A 3 x 3 pixel shuffle turns SigLIP2's 729 patches into 81 tokens, then LayerNorm and a two layer MLP map them into Orbit-Nano's embedding space. The image tokens replace 81 placeholder tokens right after the start of the first user turn.
  2. Warm up. For the first 6 % of the run only the projector learns.
  3. Joint training. Then projector and Orbit-Nano learn together, Muon on Orbit's matrices and AdamW on embeddings, head and projector, WSD schedule, sequence length 1,024.
  4. Data. FineVision subsets densefusion_1m, allava_laion, DoclingMatix, lnqa, docvqa, chartqa and textvqa (single image samples), plus 12 % text only chat (SmolTalk) against forgetting.

Limitations

  • Research preview. Single image at 384 x 384 px with 81 image tokens, so small text in documents is often misread.
  • English only. Orbit-Nano's chat ability got weaker during vision training (text validation loss 0.73 to 0.96). For text only use, take Orbit-Nano itself.
  • Can describe things that are not in the image. No safety tuning beyond the base model.

The Orbit family

Model What it does
Orbit-Nano-Compliance-E2B German and English compliance chat, rank 1 of 7 on our compliance benchmark
Orbit-Nano-Compliance-E2B-GGUF The same model for laptops
Orbit Phone Speech recognition, 2.60 % WER on LibriSpeech test-clean
Orbit-Vision Questions about images, charts and documents
Orbit-Image Pixel art from text (research preview)

License

CC-BY-NC-4.0 for the projector and the fine-tuned language model weights in this repository, until the training data licenses are fully reviewed. SigLIP2 keeps its Apache-2.0 license.

Built by Starlex AI, a Foundation AI Lab.

Downloads last month
-
Safetensors
Model size
4B params
Tensor type
F32
ยท
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for StarlexAI/orbit-vision

Finetuned
(1)
this model