README.md

metadata

title: FineRobotics Annotation Studio
emoji: 🤖
colorFrom: indigo
colorTo: blue
thumbnail: >-
  https://huggingface.co/spaces/finerobotics/annotation-studio/resolve/main/assets/studio-cover.png
sdk: docker
app_port: 7860
fullWidth: true
hf_oauth: true
hf_oauth_scopes:
  - read-repos
  - contribute-repos
  - inference-api
  - jobs
hf_oauth_expiration_minutes: 1440
short_description: Robot video → subtasks, plans and memory
license: apache-2.0
models:
  - nvidia/EgoHand-v1
  - nvidia/RT-DETR-Hand-Detector-v1
  - nvidia/SOMA-X
  - nvidia/Flora-Captioner
  - nvidia/Flora-TempSeg
  - Qwen/Qwen3.8-27B
  - Qwen/Qwen3.6-27B
  - google/gemma-4-31B-it
  - Qwen/Qwen3.8-Flash-Next
  - zai-org/GLM-5.3-Flash
  - moonshotai/Kimi-K3
  - MiniMaxAI/MiniMax-M3
  - google/gemma-4-26B-A4B-it
  - Qwen/Qwen3-VL-235B-A22B-Instruct
tags:
  - robotics
  - lerobot
  - annotation

FineRobotics Annotation Studio

Hand actions uses EgoHand with native SOMA/MHR hands and DROID-W + MoGe Small inside the existing annotation wizard. Choose a human-POV dataset camera and an HF Jobs GPU. Measured calibration comes from dataset metadata by default. An explicit experimental option estimates pinhole focal length with MoGe when calibration is absent, assuming a centered camera and no distortion. These outputs record estimated intrinsics; the calibrated accuracy reference does not apply. Aria Gen1 image rotation is explicit. The annotated copy retains every source frame, writes 156D actions and native SOMA parameters, and masks frames outside directly observed bimanual spans of at least 30 frames. This recipe has no motion infiller and needs no MANO files. Model assets come from a pinned, checksummed package embedded in Annotation Studio's public image. The Space build fetches the large weights from public upstream sources; Jobs need no model-repository access. Browsing and estimating public datasets need no login. Sign in to launch HF Jobs or access private datasets. The worker image and asset identities are verified before native inference. The same Space supplies the UI, Flora Jobs and hand-action Jobs; separate worker Spaces are unnecessary. Each hand estimate pins the currently running Studio image and rejects a later deployment until the user calculates a fresh estimate. Results link to the official lerobot/visualize_dataset.

The measured development reference is 39.41–41.04 input FPS on RTX 5090 over ten short calibrated clips, including startup, camera and native export, excluding installation/model downloads. HF GPU throughput is unmeasured. Setup, per-GPU TensorRT compilation and dataset copying add time. See the expandable pipeline and model-term details in the wizard; research outputs remain private.

Enter a LeRobot v3 dataset, choose annotations, review an estimate, then sign in with Hugging Face to run a Job using the selected annotation model. General vision models use a CPU Job and Inference Providers; Flora uses an H200 GPU Job. Jobs and inference are billed to the signed-in user's personal account. The Space itself uses free CPU hardware and has no shared HF access token.

The Space has moved from lerobot/annotation-studio to finerobotics/annotation-studio.

Execution and independent Flora subtask labeling

The worker uses the scalable lerobot.annotations.runtime engine from LeRobot #4892, with checksummed episode outputs, explicit stage dependencies, cache reuse and bounded workers. Language annotations complete before human-video generation starts, so media failures preserve the completed language output. The existing Studio controller continues to manage dataset subsets, progress, budget reservations and provider limits. Checkpoints live on the Job's disk; starting a different Job does not yet resume another Job's ephemeral disk.

Human videos directly use HumanVideoModule from LeRobot #4809, rebased onto #4892. A Studio adapter preserves the quoted first-frame-only workflow, full-resolution image edits, spending/time guards and the media viewer. End-frame editing remains available in the CLI but is not included in the Studio's single-edit quote.

In the Model selector, choose NVIDIA Flora · TempSeg + Captioner for an independent end-to-end subtask run. Flora-TempSeg predicts local intervals using NVIDIA's native 16-second windows, adaptive processing and continuation stitching. Flora-Captioner labels the reconstructed events using BEFORE/TARGET/AFTER context. Neither stage consumes Qwen3.8 predictions or existing dataset subtask labels. The selected source episode task is supplied as context; empty tasks do not trigger Qwen task derivation. Native overlapping events and gaps are retained. Both workflows save full {start, end, text} spans in each episode's annotation_studio/run.json entry, in addition to the normal Parquet language rows (which store starts). Compare the two runs on the same source revision, episodes and camera. This compares the two complete annotation workflows.

Flora uses NVIDIA's pinned native frame planner, video processor, mixed BF16/FP32 loader, greedy decoder and trained prompts. Custom Subtasks prompts must be reset before selecting Flora. The input uses the selected single robot camera, with constant-rate episode timestamps and at least 4 source FPS. Source-limited or failed captions fail explicitly; there is no fallback to Qwen.

Flora has no hosted Inference Provider route. Selecting it runs the worker on an H200 CUDA Job; the quote reserves the entire GPU timeout, including installation, weight downloads and startup. Subtasks and default numbered plans need no hosted VLM calls or token reservation. Other annotations (memory, rephrasings, interjections, visual questions, human-video prompts) and custom plan prompts still use the displayed helper VLM through Inference Providers; their cost and provider are shown only when required. Subtask timestamps and labels remain independent in these runs too. The default Qwen option keeps the CPU workflow. Flora's model card limits use to noncommercial research. Code and weight revisions are pinned in flora_captioner.py and each run manifest, including TempSeg's adapter and base revisions. This option is experimental. An initial live H200 run produced six annotated episodes; nineteen other episodes hit a duration-clock bug, now fixed. CPU tests cover native window planning, stitching, fractional-duration clips and the absence of external VLM calls. benchmark_flora.py is the older caption-only comparison on shared reference intervals, not the independent end-to-end comparison offered by the selector.

On the H200, the worker retains TempSeg and Captioner in separate native sessions. Each model is loaded and verified once, then reused across episodes; sampling, weights, precision and generation settings stay unchanged. This mode requires at least 120 GiB of GPU memory. Episode durations are rounded to the native integral- microsecond clock before scheduling, including 30-fps clips with fractional seconds.

The default tuned Qwen3.8 preset uses the action-phase prompt developed on ABC, timestamped contact sheets at 2 fps, 1,200 frames per request, and up to 80 subtasks per window. It retries failed segmentation with the original 30-second-window preset. This is an experimental prompt preset, not finetuned weights; its small held-out benchmark did not establish a quality improvement.

Models

The default is Qwen3.8 27B / Novita, with up to 32 parallel requests. Standard choices also include Qwen3.6 27B / OVHcloud and Gemma 4 31B / Novita. The selector offers nine curated VLMs from the families listed in this card. This is a practical selection, not a universal quality ranking. All nine available routes passed a live two-image JSON smoke test on 2026-09-25. Only Qwen3.8 27B has a 32-request concurrency test and the recent 33-episode all-annotation run; the new routes use a conservative eight requests, while Flash Next retains its tested two-request limit. Provider/account limits vary.

model_catalog.py owns the allowed model IDs, explicit provider routes, provider-specific generation parameters, checked pricing and context limits. Quotes and jobs pin these together. Model changes invalidate the browser quote; launch checks the selected provider is live and rejects changed catalog prices. The worker uses the pinned generation parameters, including non-Qwen models. There is no silent provider failover, so the reviewed data recipient and rates stay explicit. New model routes require a smoke test and a reviewed rate card.

Rates shown are USD per million tokens. Cached input rates are accounted for; MiniMax M3 switches to its higher pricing tier for prompts of 524,288 tokens or more. The budget reserves the worst tier and full provider context before each request, then reconciles actual reported usage. Small budgets reduce parallelism; a target too small for one conservative reservation is rejected before launch. The action-phase prompt was developed on Flash Next; quality results do not transfer automatically to other models.

Outputs

• Annotated dataset copy: a new standalone LeRobot dataset containing only successfully annotated episodes, with annotations in the standard LeRobot Parquet columns. No separate annotation JSONL files are uploaded. Every RGB camera video is trimmed to those episodes and re-encoded as H.264 (CRF 18); shared source video files are never copied wholesale. Source numeric data, tools and other annotation styles within copied episodes are retained. Indices, tasks, episode metadata and statistics are rebuilt. annotation_studio/episode_mapping.json maps new episode IDs back to the source. Original license files and the source README are retained. Failed, stopped and unselected episodes are not included. Each completed copy is published atomically with its metadata so a partial run remains loadable. Depth video is currently unsupported. Storage charges are separate from the run estimate and spending target.

Setup is split into four focused steps: Dataset, Annotations, Model, and Review. Examples and provider details expand on demand. Prompt edits and the current draft are retained when moving between steps; changing priced settings requires a new estimate.

The user chooses Private (default) or Public at review. The launch confirmation shows the output visibility. Visibility is pinned in the job manifest and applied only to the new output repository. Older manifests without a visibility field remain private.

Subtasks are generated as a dependency of plans, memory and interjections; the UI makes this explicit. Unchecked output styles are filtered from the saved annotations. All annotation types use one selected RGB video camera per run; views are not combined. Camera names load automatically from the dataset's meta/info.json before estimating, including private datasets after login. Depth and non-video features are excluded. Visual questions have a configurable cadence.

Flora execution profiles

Normal Studio jobs retain the validated reference decoder. An opt-in CUDA worker image (Dockerfile.flora) bakes in pinned Torch/Transformers and fused flash-linear-attention / causal-conv1d kernels. FLORA_EXECUTION=accelerated fails before model downloads if those kernels are unavailable. It uses bounded left-padded batches (default 4, OOM splitting), preserves sample/video order, greedy decoding, model weights, FP32 adapters/GDN parameters and native sampling. Captioner assets download while TempSeg is working; every native integrity check still runs. Both profiles now publish episode copies while annotation continues, using private Parquet slices so inference source files remain immutable.

The accelerated profile remains opt-in until a full GPU benchmark verifies runtime, memory and annotation validity. A 15-minute target for 25 ABC episodes is not a measured performance claim. CPU contract tests cannot establish CUDA numerical equivalence or the throughput of these released weights.

Human demo videos

A separate Video generation category on the Annotations step, next to the text annotations. For every subtask (or up to 1–8 per episode), the job turns the robot view into a first-person human demonstration, following LeRobot's --human_video annotation module and the video-generation stages of Zero-WAM's HumanGen pipeline:

1. the selected vision model reads the robot segment and writes the human task and an image-edit prompt;
2. Qwen/Qwen-Image-Edit-2509 removes the robot from the segment's first full-resolution frame and adds two resting human hands;
3. the vision model writes a step-by-step video prompt from the edited frame;
4. MiniMaxAI/MiniMax-H3 animates it at 480P (5 s).

Both media models run on fal through HF Inference Providers with the signed-in user's token. Selecting videos also selects Subtasks, since there is one video per subtask.

The estimate uses the selected episodes' total seconds and count, calibrated on the first video run (3 ReBot episodes, 159.7 s, 33 subtasks): 1 subtask per episode + 1 per 5.3 s, up to 1.5× that in the upper bound, with the per-episode cap applied. Each video is priced at fal's list prices: the edit at $0.03 per output megapixel (1024×1024) and the 5 s 480p video at $0.05/s, about $0.28 in total, plus about 1,750 input and 550 output VLM tokens. Video limit per episode selects all subtasks or the first 1–8 subtasks of each episode, generating one video per selected subtask. An episode with fewer subtasks produces fewer videos. Videos generated at once (1–32, default 8) limits simultaneous image edits and video generations across the whole job. Higher values can finish sooner and spend the budget faster, without changing the per-video rates. The review warns when the expected videos at that parallelism would outlast the job timeout (about 3 min per video).

Videos start only after an episode's annotations pass validation. Each request holds a conservative reservation (model_catalog.HUMAN_VIDEO) from the spending target while in flight and is reconciled to its list price when it returns. A failed video is recorded and the others continue. Videos stop at the spending target, or before the runtime limit (the job timeout when none is set), without paying for an edit whose video could not finish; the episode's annotations are kept. The launch checks that both fal mappings are live.

The output holds annotation_studio/human_videos/episode_XXXXXX/segment_XXX{.mp4, _first_frame.png,_robot_frame.png} (named by source episode) and index.json, which lists each segment's subtask, timestamps, prompts, status and materialized episode_index. These are synthetic media, not language columns. The Jobs page previews completed videos next to the robot frame, served through the Space so private outputs stay private.

Prompts and examples

Every annotation card has an Edit prompt editor with the actual model instructions, required placeholders, and a reset-to-default action. Interjections have separate opening-acknowledgement and mid-task templates. Edits are validated, kept in the browser draft, and pinned in the output run manifest. Subtask edits also apply to internal dependencies and shorter-window recovery. Other unchecked annotation types retain edits in the draft without using them in the run.

Default plans are deterministic remaining-step lists. A custom plan prompt opts into model generation at each boundary; the cost estimate includes those extra calls. Resetting it restores the deterministic default. Longer custom prompts also increase estimated input usage. Instructions can change segment counts and output lengths, so costs remain estimates.

The main screen shows two illustrative examples per annotation type. Subtask examples include timed reach/grasp/move/place phases and a continuous wiping segment, making the default granularity visible before launching a job.

Cost and privacy

Estimates are ranges, based on metadata duration, sampled frames, approximate tokens and the provider's published rates. CPU pricing is retrieved from HF. Before each request, the worker reserves a conservative token cost; successful calls reconcile with usage and uncertain failures retain their reservation. The selected total target reserves the entire job timeout's CPU cost first. This is an application spending guard, not an HF billing guarantee: provider price changes, billing differences and storage charges remain outside it. Set account billing limits on Hugging Face as an additional control.

OAuth tokens stay on the server and are passed only as HF Job secrets. They are not stored in repositories, browser storage, browser outputs or job commands. Private metadata uses temporary per-request caches. Dataset content is sent to the selected inference provider only after the user starts a job.

Reproducibility and operations

Jobs pin the Space code and source dataset revisions. pipeline-source.tar.gz contains the LeRobot source used for this deployment; pipeline-version.json records its Git commit and archive checksum. Each output includes its manifest, per-episode progress and metered inference usage. Completed shards are uploaded incrementally. A budget stop or error produces an explicitly partial result. The worker never executes code from an input dataset.

The deployment helper packages tracked LeRobot source, checks for tracked changes, tests the package checksum and uploads this directory. The existing lerobot/annotate manual-labeling Space is independent.

Local development

cd spaces/annotation-studio/frontend && bun install && bun run build
# From spaces/annotation-studio:
uv run --with-requirements requirements.txt uvicorn server:app --port 7860
uv run --no-sync pytest spaces/annotation-studio/test_studio.py
uv run --no-sync python spaces/annotation-studio/deploy.py

The Docker Space serves a React frontend and a FastAPI backend from one origin. The interface adapts visualizer design tokens and annotation colors; attribution is in THIRD_PARTY.md. OAuth uses Authorization Code with PKCE and HttpOnly session cookies. Tokens live in server memory; restarting the Space signs users out. A refreshable dashboard lists each user's jobs, progress and cancellation.

Hugging Face login is available on the deployed Space. Local mode is preview only: it cannot launch jobs using a developer's cached credentials.

The Jobs dashboard refreshes every 15 seconds and shows usage-derived cost: provider-reported input/output/cache tokens at the pinned rates, plus HF-reported CPU runtime rounded to minutes. Unknown request charges remain separately reserved; this is not an invoice. Worker usage checkpoints are saved every ten seconds. The billing link opens the user's final HF charges.

Finished annotated copies offer Show annotations, opening the first materialized episode with ?tab=annotations in the visualizer. Private results require signing in there. Older JSONL-only exports retain their Results download link.

Hand-annotated copies offer View 3D hands once an episode is materialized, opening synchronized wrists, hand skeletons and camera overlays with ?tab=hands. Operators can set HAND_VISUALIZER_URL to a compatible visualizer deployment; the default is the official LeRobot dataset visualizer.