cosmos_transfer/input_bundle/README.md

Cosmos-Transfer2.5-2B input bundle

186 frames, 1280x720, 16 FPS = 11.625 s. 186 = 2 x 93, the frame multiple the model card reports as best.

What to feed the model

File	Role
prompt.txt	text prompt (151 words; the card requires < 300)
input_video.mp4	the video being transferred: real background plate + CG robot/brick
depth.mp4	depth control -- relative inverse depth, 8-bit, near = bright
fg_mask.mp4	binary robot+brick mask, for a control's mask_path
seg.mp4	segmentation control (stable palette, see export_meta.json)
edge.mp4	Canny edge control
vis.mp4	a pre-blurred copy -- normally unused, see below
first_frame.png	real photograph of this scene at the trajectory's t=0
spec_*.json	ready-to-run parameter files (paths relative to the spec)

Run it

python examples/inference.py -i <this dir>/spec_depth_vis.json -o outputs/run1
# multi-GPU:
torchrun --nproc_per_node=4 examples/inference.py -i ... -o ...

Spec	What it does
spec_depth_vis.json	start here. depth 0.9 + vis 0.4
spec_depth_masked_vis.json	same, but depth applies only to the robot/brick
spec_depth_only.json	depth alone at 1.0, matching the shipped example
spec_multicontrol.json	all four branches

Why vis has no control_path

The repo computes the blur itself from video_path when you give vis a preset_blur_strength instead of a control video -- and that preset is, by construction, the blur the branch was trained on. vis.mp4 here was made with a hand-picked Gaussian sigma (18 px), which is a guess at that distribution. The specs therefore use the preset. vis.mp4 is kept only so you can A/B it.

Do not raise the vis weight much: it is the strongest appearance control, and pushing it up makes the output a de-blurred CG render rather than a photorealistic video. depth is where our real information is.

On masks

mask_path in this repo is binary -- white means "use this control here", black means "don't". It is not a per-pixel weight map; per-control strength is the scalar control_weight. spec_depth_masked_vis.json uses fg_mask.mp4 to confine depth to the robot and brick, which is worth trying because foreground depth is exact mesh z-buffer at full resolution while background depth is upsampled from a 320x180 capture and partly interpolated.

Two things that are easy to get wrong

Depth is not metric here. It is relative inverse depth, matching what DepthAnything (Cosmos's own extractor) produces, normalised once for the whole clip over 0.231 .. 1.688 m. Per-frame normalisation makes the control flicker and the model turns that into brightness pumping. The metric 16-bit millimetre PNGs remain in outputs/conditioning_cosmos_720p and are the source of truth; regenerate from those, never from depth.mp4.

The background is a real photograph. The capture camera is static to machine precision, so a per-pixel temporal median over the real episode footage removes the moving arm and leaves background_plate.png, pixel-aligned with the render by construction. Only the robot and the brick are CG. This is why edge.mp4 is usable at all -- the background carries real texture edges rather than point-cloud speckle.

Spec schema

Taken from the repo's own shipped examples (assets/robot_example/*/*_spec.json), not guessed: name, prompt_path, video_path, guidance, and per-control {control_path, control_weight, mask_path}. Paths are relative to the spec file.

Known limitations inherited from the render

• no_dynamics: Kinematic replay only: no mass, friction, or inertia is modelled. A grasp is a rigid attachment of the object to the gripper frame, not a contact/force simulation. Commanded speed changes only the timing of the motion, never its path.
• single_viewpoint_capture: Applies to the DEPTH channel only in this bundle. Background depth is still a single-viewpoint 2.5D capture, so surfaces the camera never saw have no points and any view far from the capture pose exposes real holes. Background appearance no longer has this limitation: rgb/vis/edge take their background from a real photograph (temporal median of the episode), not from the cloud.
• no_collision_checking: No collision checking was performed between the robot, the manipulated object(s), or the static scene, at any stage of producing this trajectory or this render.
• no_settling_at_release: SUPERSEDED for this bundle: the released object WAS settled under gravity (MuJoCo 3.11.0). Its tilt at release was 37.26 deg and 0.09 deg after settling, reached in 0.389 s of simulated time. Mass is not load-bearing: a rigid body's fall-and-settle trajectory is mass-independent, which was verified by re-running at 10x mass. Everything else in no_dynamics still holds -- the transport motion itself is kinematic.
• background depth resolution: Background depth is unprojected from a single ~320x180 capture, so it is smooth/blocky relative to the 1280x720 output. Robot and object depth comes from the mesh renderer at full output resolution and is sharp. Holes were filled by nearest valid pixel; mean 18.00%, max 20.68% of pixels per frame.