Instructions to use StarlexAI/orbit-vision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use StarlexAI/orbit-vision with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="StarlexAI/orbit-vision", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("StarlexAI/orbit-vision", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use StarlexAI/orbit-vision with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "StarlexAI/orbit-vision" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "StarlexAI/orbit-vision", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/StarlexAI/orbit-vision
- SGLang
How to use StarlexAI/orbit-vision with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "StarlexAI/orbit-vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "StarlexAI/orbit-vision", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "StarlexAI/orbit-vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "StarlexAI/orbit-vision", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use StarlexAI/orbit-vision with Docker Model Runner:
docker model run hf.co/StarlexAI/orbit-vision
Download README.md from StarlexAI/orbit-vision: direct link, hf CLI and curl.
- Browser
- Download file 4.81 kB
-
https://huggingface.co/StarlexAI/orbit-vision/resolve/main/README.md
- Command line
-
hf download hf://StarlexAI/orbit-vision/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/StarlexAI/orbit-vision/resolve/main/README.md
license: cc-by-nc-4.0
language:
- en
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- vision-language
- vlm
- orbit
- starlex
- siglip2
- muon
- research-preview
base_model:
- StarlexAI/Orbit-Nano-Compliance-E2B
- google/siglip2-so400m-patch14-384
extra_gated_heading: Orbit-Vision research preview
extra_gated_prompt: >-
Orbit-Vision is a research preview for non-commercial research. Parts of its
training data come from FineVision subsets whose licenses are still being
reviewed. By requesting access you agree to use it for non-commercial research
only.
extra_gated_fields:
Affiliation: text
Intended use: text
I agree to use this model for non-commercial research only: checkbox
Orbit-Vision v1 (research preview)
Our own language model learned to read images in a single night. 60 % on TextVQA, and only 6.5 % when the image is blank, so it really looks.
Orbit-Vision answers questions about photos, charts and documents. Its language model is Orbit-Nano, pretrained from scratch by Starlex AI. The eyes are Google's SigLIP2-SO400M image encoder, kept frozen. Starlex AI trained the projector that connects the two and fine-tuned Orbit-Nano to use it.
| Training | about 11 hours on 4 GPUs, 94,356 steps |
| Language model | Orbit-Nano, 4.03B parameters, 1.75B compute per token, pretrained by Starlex AI |
| Image encoder | SigLIP2-SO400M by Google, 384 px, frozen |
| Image tokens | 81 per image |
| License | CC-BY-NC-4.0, research preview |
Results
Held-out FineVision VQA files, never used in training, 200 questions each, greedy answers, relaxed match. The blank image column asks the same questions with an empty image. The gap shows how much of each answer comes from actually reading the image.
| Benchmark | With image | Blank image | Gain from the image |
|---|---|---|---|
| TextVQA (text in photos) | 60.0 % | 6.5 % | +53.5 points |
| ChartQA (charts) | 41.5 % | 9.5 % | +32.0 points |
| DocVQA (documents) | 35.0 % | 3.0 % | +32.0 points |
Try it
pip install torch transformers safetensors huggingface_hub pillow
python orbit_vision.py photo.jpg "What is written on the sign?"
import torch
from orbit_vision import OrbitVision
m = OrbitVision(".", device="cuda", dtype=torch.bfloat16)
print(m.ask("chart.png", "Which year has the highest value?"))
SigLIP2 is downloaded from google/siglip2-so400m-patch14-384 on first use.
How it was built
- Projector. A 3 x 3 pixel shuffle turns SigLIP2's 729 patches into 81 tokens, then LayerNorm and a two layer MLP map them into Orbit-Nano's embedding space. The image tokens replace 81 placeholder tokens right after the start of the first user turn.
- Warm up. For the first 6 % of the run only the projector learns.
- Joint training. Then projector and Orbit-Nano learn together, Muon on Orbit's matrices and AdamW on embeddings, head and projector, WSD schedule, sequence length 1,024.
- Data. FineVision subsets densefusion_1m, allava_laion, DoclingMatix, lnqa, docvqa, chartqa and textvqa (single image samples), plus 12 % text only chat (SmolTalk) against forgetting.
Limitations
- Research preview. Single image at 384 x 384 px with 81 image tokens, so small text in documents is often misread.
- English only. Orbit-Nano's chat ability got weaker during vision training (text validation loss 0.73 to 0.96). For text only use, take Orbit-Nano itself.
- Can describe things that are not in the image. No safety tuning beyond the base model.
The Orbit family
| Model | What it does |
|---|---|
| Orbit-Nano-Compliance-E2B | German and English compliance chat, rank 1 of 7 on our compliance benchmark |
| Orbit-Nano-Compliance-E2B-GGUF | The same model for laptops |
| Orbit Phone | Speech recognition, 2.60 % WER on LibriSpeech test-clean |
| Orbit-Vision | Questions about images, charts and documents |
| Orbit-Image | Pixel art from text (research preview) |
License
CC-BY-NC-4.0 for the projector and the fine-tuned language model weights in this repository, until the training data licenses are fully reviewed. SigLIP2 keeps its Apache-2.0 license.
Built by Starlex AI, a Foundation AI Lab.
