Spaces:

---

finerobotics

/

annotation-studio

Like 4

Running

Adds Human demo videos, a new Video generation category on the Annotations step next to the text annotations. For each subtask, the job generates a first-person video of human hands doing that subtask from the robot view. It mirrors LeRobot's --human_video annotation module (huggingface/lerobot#4809) and runs on the current pinned pipeline bundle, which stays unchanged.

Pipeline (per subtask, after the episode's annotations pass validation)

1. The selected VLM reads 6 robot frames and writes the human task and an image-edit prompt.
2. Qwen/Qwen-Image-Edit-2509 (fal, guidance_scale=1.5) edits the full-resolution first frame: it removes the robot and adds two resting human hands. The output is resized back to the camera size.
3. The VLM writes a step-by-step video prompt from the edited frame.
4. MiniMaxAI/MiniMax-H3 at 480P, 5 s (fal, via HF Inference Providers) animates it. The image-text-to-video task isn't in InferenceClient yet, so this reuses fal's image-to-video queue helper and adds a polling loop that respects the job's deadline.

Cost and safety

• Pricing is pinned in model_catalog.HUMAN_VIDEO: $0.04 for the edit plus $0.30 for the video (480p is $0.05/s, reserved as 6 s).
• The estimate prices every possible video (episodes × videos per episode) plus 2 VLM calls per video.
• Each media request reserves its price from the spending target before it starts, and the reservation is never refunded.
• Reaching the target or the runtime limit stops further videos but keeps the episode's annotations. A failed video is recorded and the others continue.
• The launch checks that both fal mappings are live and that the video pricing has not changed since the estimate.
• Selecting videos also selects Subtasks, and the backend enforces this.
• The token stays server-side and is sent only as the job secret. The fal CDN download goes out without auth headers.

Output

• Files are written to annotation_studio/human_videos/episode_XXXXXX/segment_XXX{.mp4,_first_frame.png,_robot_frame.png}, named by source episode.
• index.json lists each segment's subtask, timestamps, prompts and status, plus the episode_index it was materialized as. It is uploaded in the per-episode commit.
• run.json records per-episode video counts, and its usage section includes media_usd.
• The dataset card gets a "Human demonstration videos" section.

UI

• New annotation: a separate "Video generation" card with a videos-per-episode picker (1/2/3/5/8), models, price and pricing link. The cost breakdown gets a "Human demo videos" line.
• Jobs: a video count and cost line, plus a lazy preview grid (robot frame next to the human video). Media is proxied through /api/jobs/{id}/media, so private outputs stay private. That endpoint only allows the generated file paths, checks the job's owner, and never exposes prompts.

Testing

• test_human_video.py (12 tests) covers:
  • pricing and dependency validation;
  • launch checks for fal liveness and stale prices;
  • media budget accounting;
  • a real-video episode with a fake media client, checking full-resolution edits, failed segments, budget stops and the segment cap;
  • commit operations and index;
  • fal router polling, including no token sent to the CDN and the deadline guard;
  • the media endpoint's path allow-list and auth.
• The existing suite passes (86 passed). bun run build (tsc and vite) is clean.
• A real fal call through FalMedia returned a 640×480, 24 fps, 5.18 s video with a correct human hand. The edit took 14 s and the video 15 s, and the budget recorded $0.34.
• UI checked at desktop and phone width.

Not yet run: a full paid job on the deployed Space. That needs this PR merged or deployed, because jobs download the Space revision.

🤖 Generated with Claude Code