Instructions to use voyager-so/shot-reader-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use voyager-so/shot-reader-8b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="voyager-so/shot-reader-8b") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("voyager-so/shot-reader-8b") model = AutoModelForMultimodalLM.from_pretrained("voyager-so/shot-reader-8b", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use voyager-so/shot-reader-8b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "voyager-so/shot-reader-8b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "voyager-so/shot-reader-8b", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/voyager-so/shot-reader-8b
- SGLang
How to use voyager-so/shot-reader-8b with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "voyager-so/shot-reader-8b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "voyager-so/shot-reader-8b", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "voyager-so/shot-reader-8b" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "voyager-so/shot-reader-8b", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use voyager-so/shot-reader-8b with Docker Model Runner:
docker model run hf.co/voyager-so/shot-reader-8b
Shot Reader 8B

Watch a shot. Get the 3D scene.
Shot Reader reads a video shot and works out how it was made: the camera's height, angle, lens and move, who is in frame, where they stand, how they move, and the set around them. It writes all of that as one JSON shot sheet that a 3D tool or a coding agent can rebuild directly in Blender.
It's a fine-tune of Qwen3-VL-8B-Instruct, small enough to run on your own GPU.
Built by Voyager. Trained and benchmarked on Voyager Cloud, so you can run it there on an H100 in seconds.
| Shot Reader 8B | Opus 5.5 | Gemini 4 Argon | Qwen3-VL-8B (base) | |
|---|---|---|---|---|
| Camera height band and move both right (300 held-out shots) | 69% | 44% | 46% | 5%ΒΉ |
| Scene Check score, real footage (VibePrevisBench rebuilds) | 35.6 | 38.5 | 37.3 | β |
| Time per shot | 5.9 s | 14.2 s | β | β |
| Cost per shot | ~$0.002 | ~$0.06 | β | β |
ΒΉ Untrained base model, same prompt, 20 held-out shots. Opus and Gemini were given the same prompt and schema, written specifically for reverse-engineering a shot. Speed and cost are for one H100 with vLLM in FP8; bulk throughput is 0.76 s per shot, about $1.50 per million output tokens.
Why
AI can already write Blender. What it can't do is look at a reference shot and get the camera right. On the VibePrevisBench previs benchmark, agents asked for a low angle deliver eye level in up to two thirds of shots, and an orbit stays an orbit 4% of the time. Hand the same agent a plain description of the shot first, and its score jumps 15.6 points.
The models can build the scene. They need to understand it first. Shot Reader writes that understanding, straight from the video.
What people use it for
- Previs from reference. Drop in a clip and get an editable 3D blockout to relight, re-lens and reshoot. No more rebuilding shots by hand.
- Controllable AI video. Camera, layout and motion become explicit inputs you can edit, not something you hope a prompt captures.
- Searchable footage. Tag whole libraries with camera moves, framing and blocking for about $2 per thousand shots.
- Agents that build scenes. A scene-building agent gets a precise brief instead of a vague description.
Quick start
pip install torch "transformers>=4.57" pillow huggingface_hub # plus ffmpeg on your PATH
huggingface-cli download voyager-so/shot-reader-8b shot_reader.py schema.md --local-dir .
python shot_reader.py clip.mp4 \
--subject "P01: person (woman in a black polo)" \
--subject "O01: dog"
- On your own GPU: add
--load-in-4bit(needsbitsandbytes). Tested on a 16 GB RTX 5080: about 25 s for a shot, model loading included. - Fastest:
--backend vllm.
One input is one continuous shot, so cut longer videos at the edits first. Name the subjects you care about with --subject "ID: class (description)"; Shot Reader keeps your ids, so the sheet lines up with your brief.
Output
{
"shot": {"start": 0.0, "end": 5.04},
"camera": {"height_m": 1.3, "angle": "eye", "pitch_deg": -1.8, "fov_deg": 63.4,
"move": "crane", "handheld": false, "framing": "full", "subject": "P01"},
"n_subjects": 4,
"subjects": [
{"id": "P01", "class": "person", "size_m": [0.5, 0.5, 1.85], "posture": "stand",
"path": [{"t": 0.0, "pos_m": [0.0, 6.01], "facing_deg": 271.9},
{"t": 5.04, "pos_m": [0.0, 6.01], "facing_deg": 271.9}]},
{"id": "P02", "class": "person", "size_m": [0.5, 0.4, 1.66], "posture": "stand",
"path": [{"t": 0.0, "pos_m": [-0.9, 6.38], "facing_deg": 15.1},
{"t": 5.04, "pos_m": [3.1, 7.5], "facing_deg": 15.1}]},
{"id": "crate_1", "class": "crate", "size_m": [0.82, 0.41, 0.72], "path": [{"t": 0.0, "pos_m": [-3.5, 11.6]}]},
{"id": "crate_2", "class": "crate", "size_m": [0.75, 0.6, 0.62], "path": [{"t": 0.0, "pos_m": [1.28, 7.28]}]}
],
"n_events": 0, "events": [],
"environment": {"type": "interior", "indoor": true, "desc": "workshop room"},
"n_set": 3,
"set": [{"class": "wall", "pos_m": [-0.1, 11.11], "size_m": [11.28, 0.1, 3.2], "yaw_deg": 278.1}, "..."]
}
That's the real output for the shot in the video, run in 4-bit on an RTX 5080.
Positions are metres on the ground plane, with the origin under the camera in the first frame, +x to the right and +y forward. The full field reference is in schema.md.
Using it directly
The model sees 16 frames sampled evenly across the shot, resized to 448Γ256, followed by the prompt in shot_reader.py (PROMPT plus schema.md). Use the same frames and prompt for the results above; shot_reader.py is a single file you can copy from.
Training
- Base model: Qwen3-VL-8B-Instruct, LoRA r64 on the language model's attention and MLP layers, merged into these weights. The adapter alone is in
lora/. - Synthetic shots: thousands of shots rendered from procedurally generated scenes, where every camera, position and path is known exactly. Some were re-rendered photoreal with a video model.
- Real footage: public-domain footage from Wikimedia Commons and the Prelinger Archives, labelled by a frontier model.
- Output format: the model was trained to write the camera and subjects first, then typed events (jump, sit, turn, ...), then the set.
No benchmark tasks were used for training.
Limitations
- Events and routes: these are its weakest fields on real footage. Opus 5.5 still writes richer action events.
- Positions: estimates from a single camera, typically within a few metres on real footage.
- Input: one continuous shot at a time.
License
Apache 2.0, the same as the base model.
Links
- Voyager: https://voyager.so
- Run it on Voyager Cloud: https://voyager.so/cloud
Citation
The previs benchmark that motivated this model:
@article{vibeprevisbench2026,
title = {VibePrevisBench},
journal = {arXiv preprint arXiv:2611.15789},
year = {2026},
url = {https://arxiv.org/abs/2611.15789}
}
- Downloads last month
- 61
Model tree for voyager-so/shot-reader-8b
Base model
Qwen/Qwen3-VL-8B-Instruct