Image-Text-to-Text
Transformers
Safetensors
English
orbit_nano
text-generation
vision-language
vlm
orbit
starlex
siglip2
muon
research-preview
conversational
custom_code
Instructions to use StarlexAI/orbit-vision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use StarlexAI/orbit-vision with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="StarlexAI/orbit-vision", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("StarlexAI/orbit-vision", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use StarlexAI/orbit-vision with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "StarlexAI/orbit-vision" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "StarlexAI/orbit-vision", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/StarlexAI/orbit-vision
- SGLang
How to use StarlexAI/orbit-vision with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "StarlexAI/orbit-vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "StarlexAI/orbit-vision", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "StarlexAI/orbit-vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "StarlexAI/orbit-vision", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use StarlexAI/orbit-vision with Docker Model Runner:
docker model run hf.co/StarlexAI/orbit-vision
|
Download README.md from StarlexAI/orbit-vision: direct link, hf CLI and curl.
- Browser
- Download file 4.81 kB
-
https://huggingface.co/StarlexAI/orbit-vision/resolve/main/README.md
- Command line
-
hf download hf://StarlexAI/orbit-vision/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/StarlexAI/orbit-vision/resolve/main/README.md
4.81 kB
| license: cc-by-nc-4.0 | |
| language: [en] | |
| library_name: transformers | |
| pipeline_tag: image-text-to-text | |
| tags: [vision-language, vlm, orbit, starlex, siglip2, muon, research-preview] | |
| base_model: [StarlexAI/Orbit-Nano-Compliance-E2B, google/siglip2-so400m-patch14-384] | |
| extra_gated_heading: Orbit-Vision research preview | |
| extra_gated_prompt: >- | |
| Orbit-Vision is a research preview for non-commercial research. Parts of its training data come from FineVision | |
| subsets whose licenses are still being reviewed. By requesting access you agree to use it for non-commercial | |
| research only. | |
| extra_gated_fields: | |
| Affiliation: text | |
| Intended use: text | |
| I agree to use this model for non-commercial research only: checkbox | |
| # Orbit-Vision v1 (research preview) | |
| **Our own language model learned to read images in a single night. 60 % on TextVQA, and only 6.5 % when the image | |
| is blank, so it really looks.** | |
|  | |
| Orbit-Vision answers questions about photos, charts and documents. Its language model is | |
| [Orbit-Nano](https://huggingface.co/StarlexAI/Orbit-Nano-Compliance-E2B), pretrained from scratch by Starlex AI. The | |
| eyes are Google's SigLIP2-SO400M image encoder, kept frozen. Starlex AI trained the projector that connects the two | |
| and fine-tuned Orbit-Nano to use it. | |
| | | | | |
| |---|---| | |
| | Training | about 11 hours on 4 GPUs, 94,356 steps | | |
| | Language model | Orbit-Nano, 4.03B parameters, 1.75B compute per token, pretrained by Starlex AI | | |
| | Image encoder | SigLIP2-SO400M by Google, 384 px, frozen | | |
| | Image tokens | 81 per image | | |
| | License | CC-BY-NC-4.0, research preview | | |
| ## Results | |
| Held-out FineVision VQA files, never used in training, 200 questions each, greedy answers, relaxed match. The blank | |
| image column asks the same questions with an empty image. The gap shows how much of each answer comes from actually | |
| reading the image. | |
| | Benchmark | With image | Blank image | Gain from the image | | |
| |---|---|---|---| | |
| | TextVQA (text in photos) | **60.0 %** | 6.5 % | +53.5 points | | |
| | ChartQA (charts) | **41.5 %** | 9.5 % | +32.0 points | | |
| | DocVQA (documents) | **35.0 %** | 3.0 % | +32.0 points | | |
| ## Try it | |
| ```bash | |
| pip install torch transformers safetensors huggingface_hub pillow | |
| python orbit_vision.py photo.jpg "What is written on the sign?" | |
| ``` | |
| ```python | |
| import torch | |
| from orbit_vision import OrbitVision | |
| m = OrbitVision(".", device="cuda", dtype=torch.bfloat16) | |
| print(m.ask("chart.png", "Which year has the highest value?")) | |
| ``` | |
| SigLIP2 is downloaded from [google/siglip2-so400m-patch14-384](https://huggingface.co/google/siglip2-so400m-patch14-384) | |
| on first use. | |
| ## How it was built | |
| 1. **Projector.** A 3 x 3 pixel shuffle turns SigLIP2's 729 patches into 81 tokens, then LayerNorm and a two layer | |
| MLP map them into Orbit-Nano's embedding space. The image tokens replace 81 placeholder tokens right after the | |
| start of the first user turn. | |
| 2. **Warm up.** For the first 6 % of the run only the projector learns. | |
| 3. **Joint training.** Then projector and Orbit-Nano learn together, Muon on Orbit's matrices and AdamW on | |
| embeddings, head and projector, WSD schedule, sequence length 1,024. | |
| 4. **Data.** FineVision subsets densefusion_1m, allava_laion, DoclingMatix, lnqa, docvqa, chartqa and textvqa | |
| (single image samples), plus 12 % text only chat (SmolTalk) against forgetting. | |
| ## Limitations | |
| - Research preview. Single image at 384 x 384 px with 81 image tokens, so small text in documents is often misread. | |
| - English only. Orbit-Nano's chat ability got weaker during vision training (text validation loss 0.73 to 0.96). | |
| For text only use, take [Orbit-Nano](https://huggingface.co/StarlexAI/Orbit-Nano-Compliance-E2B) itself. | |
| - Can describe things that are not in the image. No safety tuning beyond the base model. | |
| ## The Orbit family | |
| | Model | What it does | | |
| |---|---| | |
| | [Orbit-Nano-Compliance-E2B](https://huggingface.co/StarlexAI/Orbit-Nano-Compliance-E2B) | German and English compliance chat, rank 1 of 7 on our compliance benchmark | | |
| | [Orbit-Nano-Compliance-E2B-GGUF](https://huggingface.co/StarlexAI/Orbit-Nano-Compliance-E2B-GGUF) | The same model for laptops | | |
| | [Orbit Phone](https://huggingface.co/StarlexAI/orbit-phone) | Speech recognition, 2.60 % WER on LibriSpeech test-clean | | |
| | [Orbit-Vision](https://huggingface.co/StarlexAI/orbit-vision) | Questions about images, charts and documents | | |
| | [Orbit-Image](https://huggingface.co/StarlexAI/orbit-image) | Pixel art from text (research preview) | | |
| ## License | |
| CC-BY-NC-4.0 for the projector and the fine-tuned language model weights in this repository, until the training data | |
| licenses are fully reviewed. SigLIP2 keeps its Apache-2.0 license. | |
| Built by Starlex AI, a Foundation AI Lab. |