Instructions to use nvidia/Flora-Captioner with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/Flora-Captioner with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="nvidia/Flora-Captioner")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("nvidia/Flora-Captioner")
model = AutoModelForMultimodalLM.from_pretrained("nvidia/Flora-Captioner", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/Flora-Captioner with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/Flora-Captioner"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Flora-Captioner",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/Flora-Captioner

SGLang

How to use nvidia/Flora-Captioner with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/Flora-Captioner" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Flora-Captioner",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/Flora-Captioner" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Flora-Captioner",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use nvidia/Flora-Captioner with Docker Model Runner:
```
docker model run hf.co/nvidia/Flora-Captioner
```

Flora-Captioner / batch_utils /README.md

Omar L

Document third-party inference dependencies

e3a9677 verified 3 days ago

preview code

Raw History Blame

4.34 kB

# FLORA Data Engine inference

This directory contains the complete **TempSeg → Captioner** workflow, identical
in both model repositories. Use either copy. TempSeg predicts action intervals;
the Captioner describes those predicted intervals. The native windowing,
adaptive retries, stitching, prompts, source timestamps, and camera layouts are
included; no internal source checkout or running demo is required.

## Setup

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use. See [third-party notices](THIRD_PARTY_NOTICES.md).

Python 3.12 and a CUDA GPU with 80 GB VRAM are the validated configuration.
The two models run sequentially. Download/cache space of at least 150 GB is
recommended for the released FP32 Captioner, base model, and adapter.

From a downloaded copy of either repository:

```bash
python -m pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r batch_utils/requirements.txt
hf auth login
```

Both private model repositories must be accessible to the logged-in account.
`HF_TOKEN` is also supported; do not put a token in source code or command-line
arguments. Weights are fetched automatically at the revisions in `models.json`.
Set `FLORA_MODEL_CACHE` to choose the local model directory.

The NVIDIA repositories are selected by default. To override them:

```bash
export FLORA_TEMPSEG_REPO="nvidia/Flora-TempSeg"
export FLORA_CAPTIONER_REPO="nvidia/Flora-Captioner"
```

## Single video

```bash
python infer.py --video videos/human.mp4 --profile human --output human.json
```

Three synchronized robot cameras:

```bash
python infer.py --video videos/top.mp4 \
  --left-wrist videos/left.mp4 --right-wrist videos/right.mp4 \
  --profile yam --output yam.json
```

Use `--profile dex` for Dex or `--profile tianji` for Tianji inputs.
YAM defaults to native 16-second temporal windows; use `--yam32` for the native
32-second YAM profile. Human uses one video. Robot profiles accept one view or
all three; `--yam32` requires all three. Wrist files must share the main video's
relative frame timestamps, frame count, and duration. Inputs are not resampled.

The validated input limits are 120 seconds, 256 MiB per file, 4096 pixels per
side, 7200 frames, and at least 4 source FPS. Variable-rate video is supported
by the single-view Human path. `--instruction` optionally supplies an episode
task; leave it blank for the default workflow.

CPU-only media validation performs no inference or model downloads:

```bash
python -m batch_utils --video videos/human.mp4 --profile human --validate-only
```

## JSONL jobs

Each row specifies one job. Camera paths are relative to the JSONL file:

```json
{"id":"human01","video":"videos/human.mp4","profile":"human"}
{"id":"robot01","video":"videos/top.mp4","left_wrist":"videos/left.mp4","right_wrist":"videos/right.mp4","profile":"yam","yam32":true}
```

```bash
python infer.py --input-jsonl jobs.jsonl --output-dir results
```

Jobs and predicted segments run sequentially; JSONL is not GPU batch scheduling.
Existing outputs are preserved unless `--overwrite` is explicit. An error stops
the command; successfully written earlier jobs remain available.

## Python usage

```python
from batch_utils import DataEnginePipeline
engine = DataEnginePipeline()
result = engine.annotate("videos/human.mp4", profile="human")
for segment in result["segments"]:
    print(segment["start_s"], segment["end_s"], segment["caption"])
```

Outputs contain clip-relative seconds, caption completion status, camera roles,
input SHA-256, and model revisions. Native overlapping intervals are retained.
`include_audit=True` / `--include-audit` adds temporal and caption-input records.
The code preserves FP32 LoRA/GDN values and validates checkpoint manifests.
Full manifest verification happens on first use in the process; reuse requires
unchanged manifest bytes and all file identities/timestamps. FP32 parameter
values are checked on every load. First-time downloads and verification are
substantial; later jobs reuse the verified local files.

Example output (illustrative):

```json
{"segments":[{"start_s":0.8,"end_s":3.2,"caption":"Pick up the cup with the right hand.","status":"complete"}]}
```

	# FLORA Data Engine inference

	This directory contains the complete TempSeg → Captioner workflow, identical
	in both model repositories. Use either copy. TempSeg predicts action intervals;
	the Captioner describes those predicted intervals. The native windowing,
	adaptive retries, stitching, prompts, source timestamps, and camera layouts are
	included; no internal source checkout or running demo is required.

	## Setup

	This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use. See [third-party notices](THIRD_PARTY_NOTICES.md).

	Python 3.12 and a CUDA GPU with 80 GB VRAM are the validated configuration.
	The two models run sequentially. Download/cache space of at least 150 GB is
	recommended for the released FP32 Captioner, base model, and adapter.

	From a downloaded copy of either repository:

	```bash
	python -m pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu128
	python -m pip install -r batch_utils/requirements.txt
	hf auth login
	```

	Both private model repositories must be accessible to the logged-in account.
	`HF_TOKEN` is also supported; do not put a token in source code or command-line
	arguments. Weights are fetched automatically at the revisions in `models.json`.
	Set `FLORA_MODEL_CACHE` to choose the local model directory.

	The NVIDIA repositories are selected by default. To override them:

	```bash
	export FLORA_TEMPSEG_REPO="nvidia/Flora-TempSeg"
	export FLORA_CAPTIONER_REPO="nvidia/Flora-Captioner"
	```

	## Single video

	```bash
	python infer.py --video videos/human.mp4 --profile human --output human.json
	```

	Three synchronized robot cameras:

	```bash
	python infer.py --video videos/top.mp4 \
	--left-wrist videos/left.mp4 --right-wrist videos/right.mp4 \
	--profile yam --output yam.json
	```

	Use `--profile dex` for Dex or `--profile tianji` for Tianji inputs.
	YAM defaults to native 16-second temporal windows; use `--yam32` for the native
	32-second YAM profile. Human uses one video. Robot profiles accept one view or
	all three; `--yam32` requires all three. Wrist files must share the main video's
	relative frame timestamps, frame count, and duration. Inputs are not resampled.

	The validated input limits are 120 seconds, 256 MiB per file, 4096 pixels per
	side, 7200 frames, and at least 4 source FPS. Variable-rate video is supported
	by the single-view Human path. `--instruction` optionally supplies an episode
	task; leave it blank for the default workflow.

	CPU-only media validation performs no inference or model downloads:

	```bash
	python -m batch_utils --video videos/human.mp4 --profile human --validate-only
	```

	## JSONL jobs

	Each row specifies one job. Camera paths are relative to the JSONL file:

	```json
	{"id":"human01","video":"videos/human.mp4","profile":"human"}
	{"id":"robot01","video":"videos/top.mp4","left_wrist":"videos/left.mp4","right_wrist":"videos/right.mp4","profile":"yam","yam32":true}
	```

	```bash
	python infer.py --input-jsonl jobs.jsonl --output-dir results
	```

	Jobs and predicted segments run sequentially; JSONL is not GPU batch scheduling.
	Existing outputs are preserved unless `--overwrite` is explicit. An error stops
	the command; successfully written earlier jobs remain available.

	## Python usage

	```python
	from batch_utils import DataEnginePipeline

	engine = DataEnginePipeline()
	result = engine.annotate("videos/human.mp4", profile="human")
	for segment in result["segments"]:
	print(segment["start_s"], segment["end_s"], segment["caption"])
	```

	Outputs contain clip-relative seconds, caption completion status, camera roles,
	input SHA-256, and model revisions. Native overlapping intervals are retained.
	`include_audit=True` / `--include-audit` adds temporal and caption-input records.
	The code preserves FP32 LoRA/GDN values and validates checkpoint manifests.
	Full manifest verification happens on first use in the process; reuse requires
	unchanged manifest bytes and all file identities/timestamps. FP32 parameter
	values are checked on every load. First-time downloads and verification are
	substantial; later jobs reuse the verified local files.

	Example output (illustrative):

	```json
	{"segments":[{"start_s":0.8,"end_s":3.2,"caption":"Pick up the cup with the right hand.","status":"complete"}]}
	```