FastH3-Omni: Gradio 6 reference composer UI

README.md

@@ -1,24 +1,38 @@
 ---
-title: FastH3 OmniRef · FP8 (benchmark)
+title: FastH3-Omni
 emoji: 🎬
 colorFrom: blue
 colorTo: indigo
 sdk: gradio
-sdk_version: 5.32.0
+sdk_version: 6.30.0
 app_file: app.py
 python_version: "3.12"
 pinned: false
-short_description: FP8 variant of FastH3 OmniRef, for ZeroGPU benchmarking
+short_description: FastVideo 8-step video+audio from image/video/audio refs
 startup_duration_timeout: 1h
 models:
-  - FastVideo/FastH3-OmniRef-FP8
+  - FastVideo/FastVideo-FastH3-Omni-8-Step-V1
+  - FastVideo/FastH3-Omni-Comfy
+  - FastVideo/FastH3-Omni-FP8
+  - FastVideo/FastH3-Omni-NVFP4-QAD
+  - FastVideo/FastH3-NVFP4-VAE-Decoder
+  - FastVideo/FastH3-Omni-FP8-Comfy
+  - FastVideo/FastH3-Omni-NVFP4-QAD-Comfy
   - MiniMaxAI/MiniMax-H3
 ---
 
-# FastH3 OmniRef · FP8 variant (ZeroGPU benchmark)
+# FastH3-Omni
 
-A duplicate of `diffusers-internal-dev/fasth3-omniref` that runs the **FP8 W8A8**
-OmniRef export (`FastVideo/FastH3-OmniRef-FP8`) instead of the packed NVFP4
-export, to compare cold-start and warm generation time on the RTX PRO 6000
-ZeroGPU slice. Both variants run the 128-token VSA tiles through the bf16 Triton
-emulator (the FP4 sparse-attention kernel is absent from the published wheel).
+[FastVideo's 8-step FastH3-Omni](https://huggingface.co/FastVideo/FastVideo-FastH3-Omni-8-Step-V1) student of MiniMax-H3 on ZeroGPU: ordered image, video and audio references plus
+a prompt become a short clip with a matching soundtrack. Runs the FP8 W8A8 export (`FastVideo/FastH3-Omni-FP8`)
+with the 128-token VSA tiles on the bf16 Triton emulator.
+
+The prompt composer is a custom `gr.HTML` component (`composer.html` / `composer.css` / `composer.js`): references
+get `<Picture n>` / `<Video n>` / `<Audio n>` tags as they are added, and the prompt is either assembled by the
+six-section H3 builder or written by hand with those tags. "Upsample" rewrites it through
+`hugging-apps/minimax-h3-prompt-rewriter`.
+
+FastVideo is installed from a wheel of `hao-ai-lab/FastVideo@376e445f` whose unused `gradio==5.32.0` pin is relaxed,
+so the Space runs Gradio 6.
+
+All checkpoints: [FastH3-Omni collection](https://huggingface.co/collections/FastVideo/fasth3-omni-6ad501acfccdebd3c3b7445b).

app.py

@@ -1,17 +1,16 @@
-"""FastH3 OmniRef · reference-to-video (+audio) on ZeroGPU.
+"""FastH3-Omni · reference-to-video (+audio) on ZeroGPU.
 
-Give one or more reference images (and optionally a reference audio clip),
-describe the subject and the scene, and the FastVideo FastH3 OmniRef student
-generates a short video with a matching 32 kHz soundtrack in eight fused forwards.
-
-Prompts are assembled into the model's six-section H3 format
-(subject_definitions / summary / retention_analysis / detailed_description /
-overall_soundscape / non_diegetic_music), the form the checkpoint was trained on.
-See fasth3_engine.py for the ZeroGPU + FastVideo integration notes.
+Ordered image, video and audio references plus a prompt become a short video
+with a matching 32 kHz soundtrack in eight fused forwards of the FastVideo
+FastH3 OmniRef student. A plain prompt is wrapped into the model's six-section
+H3 format; "Upsample prompt" rewrites it with the MiniMax-H3 prompt rewriter
+instead. See fasth3_engine.py for the ZeroGPU + FastVideo integration notes.
 """
 import os
 import tempfile
 import threading
+from functools import cache
+from pathlib import Path
 
 import gradio as gr
 from PIL import Image
@@ -33,64 +32,118 @@ except ImportError:  # local / CI
 
 from fasth3_engine import FastH3OmniRefEngine, MODEL_REPO, prefetch
 
-# --- resolution / length presets -------------------------------------------
-# Width/height must be multiples of 32; frame counts must be 17n + 5. The model
-# card validates at 768p, so that is the default; 480p is the fast option.
-RESOLUTIONS = {
-    "Landscape · 1344×768 (recommended)": (1344, 768),
-    "Portrait · 768×1344 (recommended)": (768, 1344),
-    "Landscape · 832×480 (fast)": (832, 480),
-    "Portrait · 480×832 (fast)": (480, 832),
-}
-DEFAULT_RES = "Landscape · 1344×768 (recommended)"
-# The model accepts 124-362 frames (5-15 s at 24 fps); frame counts align to 17n+5.
-LENGTHS = {
-    "Standard · ~5.2s (124f)": 124,
-    "Medium · ~6.6s (158f)": 158,
-    "Long · ~8s (192f)": 192,
+REWRITER_SPACE = os.getenv("FASTH3_REWRITER", "hugging-apps/minimax-h3-prompt-rewriter")
+
+CANVASES = {
+    "1344x768 · 16:9 recommended": (768, 1344),
+    "832x480 · 16:9 fast": (480, 832),
+    "1280x704 · 16:9": (704, 1280),
+    "960x544 · 16:9": (544, 960),
+    "768x1344 · 9:16 recommended": (1344, 768),
+    "480x832 · 9:16 fast": (832, 480),
+    "768x768 · 1:1": (768, 768),
+    "1024x1024 · 1:1": (1024, 1024),
+    "1024x768 · 4:3": (768, 1024),
+    "768x1024 · 3:4": (1024, 768),
+    "1536x672 · 21:9": (672, 1536),
 }
-DEFAULT_LEN = "Standard · ~5.2s (124f)"
-
-DEFAULT_SUBJECT = "a person in a casual knit sweater"
-DEFAULT_SCENE = ("sits on a sofa in a warm, lamp-lit living room, looking at the camera and "
-                 "speaking calmly, with small natural head and hand movements.")
-DEFAULT_SOUND = "soft indoor room tone and the person's clear, calm speaking voice"
-DEFAULT_MUSIC = "none"
-
-
-def build_h3_prompt(subject: str, scene: str, soundscape: str, music: str, n_imgs: int) -> str:
-    """Assemble the six-section H3 prompt the checkpoint was trained on."""
-    subject = (subject or "the reference subject").strip().rstrip(".")
-    scene = (scene or "").strip()
-    pics = " and ".join(f"<Picture {i + 1}>" for i in range(max(1, n_imgs)))
-    sound = (soundscape or "").strip().rstrip(".") or "natural ambient sound that matches the scene"
-    mus = (music or "").strip().rstrip(".") or "none"
+DEFAULT_CANVAS = "1344x768 · 16:9 recommended"
+PORTRAIT_CANVAS = "768x1344 · 9:16 recommended"
+FPS, FRAMES_PER_CHUNK, LATENTS_PER_CHUNK = 24, 17, 5
+MIN_FRAMES, MAX_FRAMES = 124, 362
+MIN_DURATION, MAX_UI_DURATION = 5, 14
+MIN_REFERENCE_VIDEO, MAX_REFERENCE_VIDEO = 2.0, 15.0
+MAX_IMAGE_SLOTS, OPEN_IMAGE_SLOTS = 9, 2
+REWRITER_IMAGE_SLOTS = 4
+DEFAULT_PIXEL_FRAMES = 1344 * 768 * 124
+
+APP_DIR = Path(__file__).resolve().parent
+EXAMPLES_DIR = APP_DIR / "examples"
+LIMITS = {"image": 9, "video": 3, "audio": 3}
+REWRITER_LIMITS = {"image": 4, "video": 1, "audio": 1}
+
+EXAMPLES = [
+    {
+        "title": "Beach promenade at golden hour",
+        "image": "woman.jpg",
+        "desc": "a young woman with a dark curly afro and gold hoop earrings, wearing a white t-shirt",
+        "scene": "walks along a sunlit beach promenade at golden hour, smiling and glancing toward the camera, "
+        "palm trees and the ocean behind her.",
+        "sound": "ocean waves, a gentle breeze and distant seagulls",
+        "music": "none",
+    },
+    {
+        "title": "Dog bounding through snow",
+        "image": "dog.jpg",
+        "desc": "a pale cream, fluffy long-haired dog with floppy ears and a dark nose",
+        "scene": "bounds across a snowy field in bright winter sunlight, kicking up powder snow, then shakes itself off.",
+        "sound": "crunching snow, happy panting and a cold wind",
+        "music": "none",
+    },
+    {
+        "title": "Birthday cake with candles",
+        "image": "cake.jpg",
+        "desc": "a whole chocolate layer cake with chocolate frosting, pastel sprinkles and chocolate shards on top",
+        "scene": "sits at the center of a festive birthday table in a warm, dim room as lit candles flicker and "
+        "colorful confetti drifts down around it.",
+        "sound": "soft party chatter and a faint crackle",
+        "music": "cheerful, upbeat birthday music",
+    },
+]
+
+
+def composer_examples():
+    rows = []
+    for example in EXAMPLES:
+        path = str(EXAMPLES_DIR / example["image"])
+        rows.append({
+            "title": example["title"],
+            "refs": [{"kind": "image", "path": path, "url": f"/gradio_api/file={path}", "name": example["image"],
+                      "desc": example["desc"]}],
+            "builder": {"scene": example["scene"], "sound": example["sound"], "music": example["music"],
+                        "style": "realistic photographic"},
+        })
+    return rows
+
+
+def snap_frames(seconds: float) -> int:
+    frames = max(1, round(float(seconds) * FPS))
+    while frames % FRAMES_PER_CHUNK != LATENTS_PER_CHUNK:
+        frames += 1
+    return frames
+
+
+def reference_labels(references):
+    names = {"image": "Picture", "video": "Video", "audio": "Audio"}
+    counts, labels = {}, []
+    for kind, _ in references:
+        counts[kind] = counts.get(kind, 0) + 1
+        labels.append((kind, f"<{names[kind]} {counts[kind]}>"))
+    return labels
+
+
+def to_h3_prompt(prompt: str, references) -> str:
+    """Wrap a plain prompt into the six-section H3 format the checkpoint was trained on."""
+    prompt = (prompt or "").strip()
+    if "subject_definitions:" in prompt:
+        return prompt
+    labels = reference_labels(references)
+    visual = " and ".join(label for kind, label in labels if kind != "audio") or "the references"
+    audio = [label for kind, label in labels if kind == "audio"]
+    sound = (f"the voice and sound of {' and '.join(audio)}, in a natural acoustic space matching the scene"
+             if audio else "natural ambient sound that matches the scene")
     return (
-        f"subject_definitions:\n<Subject 1> is {subject}, shown in {pics}.\n\n"
-        f"summary:\nA short, realistic live-action video of {subject}.\n\n"
+        f"subject_definitions:\n<Subject 1> is the subject shown in {visual}.\n\n"
+        f"summary:\n{prompt}\n\n"
         f"retention_analysis:\n<Subject 1>: fully_preserved - the reference subject's identity, "
         f"appearance and details are kept.\n\n"
         f"detailed_description:\nThe target video is in a realistic photographic style.\n"
-        f"[Shot 1] <Subject 1> {scene}\n\n"
+        f"[Shot 1] {prompt}\n\n"
         f"overall_soundscape:\n{sound}.\n\n"
-        f"non_diegetic_music:\n{mus}."
+        f"non_diegetic_music:\nnone."
     )
 
 
-# Structured examples (subject, scene, soundscape, music, image) — they inherit
-# the recommended 768p resolution from DEFAULT_RES.
-EXAMPLES = [
-    ["a young woman with a dark curly afro and gold hoop earrings, wearing a white t-shirt",
-     "walks along a sunlit beach promenade at golden hour, smiling and glancing toward the camera, palm trees and the ocean behind her.",
-     "ocean waves, a gentle breeze and distant seagulls", "none", "examples/woman.jpg"],
-    ["a pale cream, fluffy long-haired dog with floppy ears and a dark nose",
-     "bounds across a snowy field in bright winter sunlight, kicking up powder snow, then shakes itself off.",
-     "crunching snow, happy panting and a cold wind", "none", "examples/dog.jpg"],
-    ["a whole chocolate layer cake with chocolate frosting, pastel sprinkles and chocolate shards on top",
-     "sits at the center of a festive birthday table in a warm, dim room as lit candles flicker and colorful confetti drifts down around it.",
-     "soft party chatter and a faint crackle", "cheerful, upbeat birthday music", "examples/cake.jpg"],
-]
-
 # --- engine (built lazily inside the first GPU window) ----------------------
 _ENGINE: FastH3OmniRefEngine | None = None
 _ENGINE_LOCK = threading.Lock()
@@ -116,11 +169,11 @@ def _warmup():
     if the startup GPU isn't granted this raises and we fall back to lazy."""
     eng = get_engine()
     try:
-        w, h = RESOLUTIONS[DEFAULT_RES]
-        img = os.path.join(os.path.dirname(os.path.abspath(__file__)), "examples", "woman.jpg")
-        prompt = build_h3_prompt("the reference subject", "stands still and looks at the camera",
-                                 "quiet room tone", "none", 1)
-        eng.generate(prompt, [img], None, h, w, LENGTHS[DEFAULT_LEN], 0)
+        h, w = CANVASES[DEFAULT_CANVAS]
+        img = str(EXAMPLES_DIR / "woman.jpg")
+        references = [("image", img)]
+        prompt = to_h3_prompt("The person stands still and looks at the camera. Quiet room tone.", references)
+        eng.generate(prompt, references, h, w, MIN_FRAMES, 0)
         print("[warmboot] warmup generation complete (FP4 JIT + shape warmed)", flush=True)
     except Exception as e:
         print(f"[warmboot] warmup generation skipped ({e!r}); JIT will hit first call", flush=True)
@@ -139,139 +192,280 @@ else:
     threading.Thread(target=prefetch, daemon=True).start()
 
 
-def _save_image(arr_or_path, idx: int) -> str | None:
-    if arr_or_path is None:
-        return None
-    tmp = tempfile.NamedTemporaryFile(suffix=f"_ref{idx}.png", delete=False)
-    if isinstance(arr_or_path, str):
-        Image.open(arr_or_path).convert("RGB").save(tmp.name)
-    else:
-        Image.fromarray(arr_or_path).convert("RGB").save(tmp.name)
+def probe(path: str):
+    import av
+
+    def seconds(stream, container):
+        if stream.duration is not None and stream.time_base is not None:
+            return float(stream.duration * stream.time_base)
+        return None if container.duration is None else container.duration / av.time_base
+
+    with av.open(path) as container:
+        video = seconds(container.streams.video[0], container) if container.streams.video else None
+        audio = seconds(container.streams.audio[0], container) if container.streams.audio else None
+    return video, audio
+
+
+def _allowed(path: str) -> bool:
+    from gradio.utils import get_upload_folder
+
+    real = os.path.realpath(path)
+    roots = [os.path.realpath(get_upload_folder()), os.path.realpath(EXAMPLES_DIR)]
+    return any(real == root or real.startswith(root + os.sep) for root in roots) and os.path.isfile(real)
+
+
+def references_from(composer: dict):
+    refs = (composer or {}).get("refs") or []
+    if any(ref.get("uploading") for ref in refs):
+        raise gr.Error("A reference is still uploading. Give it a second and try again.")
+    for ref in refs:
+        if ref.get("error"):
+            raise gr.Error(ref["error"])
+    references = []
+    for ref in refs:
+        kind, path = ref.get("kind"), ref.get("path")
+        if kind not in LIMITS or not path or not _allowed(path):
+            raise gr.Error("One of the references could not be read. Remove it and add it again.")
+        references.append((kind, _as_rgb_png(path, len(references)) if kind == "image" else path))
+    for kind, limit in LIMITS.items():
+        if sum(k == kind for k, _ in references) > limit:
+            raise gr.Error(f"At most {limit} {kind} references.")
+    return references
+
+
+def _as_rgb_png(path: str, index: int) -> str:
+    tmp = tempfile.NamedTemporaryFile(suffix=f"_ref{index + 1}.png", delete=False)
+    Image.open(path).convert("RGB").save(tmp.name)
     return tmp.name
 
 
-def _gpu_duration(*args, **kwargs):
+def soundtrack_seconds(references):
+    carried = []
+    for kind, path in references:
+        if kind == "image":
+            continue
+        _, audio_seconds = probe(path)
+        if audio_seconds is not None:
+            carried.append(audio_seconds)
+    return carried
+
+
+def check(prompt: str, references) -> None:
+    if not references:
+        raise gr.Error("Add at least one image or video reference.")
+    if not any(kind != "audio" for kind, _ in references):
+        raise gr.Error("Audio needs an image or a video alongside it.")
+    if not prompt or not prompt.strip():
+        raise gr.Error("Describe the video: fill in the builder or write a prompt.")
+    if "<removed>" in prompt:
+        raise gr.Error("The prompt still refers to a removed reference (<removed>). Edit it out first.")
+    for kind, path in references:
+        if kind != "video":
+            continue
+        video_seconds, _ = probe(path)
+        if video_seconds is None:
+            raise gr.Error("A reference video has no video stream. Add it as audio instead.")
+        if not MIN_REFERENCE_VIDEO <= video_seconds <= MAX_REFERENCE_VIDEO:
+            raise gr.Error(
+                f"A reference video is {video_seconds:.1f} s. Use clips between "
+                f"{MIN_REFERENCE_VIDEO:g} and {MAX_REFERENCE_VIDEO:g} seconds."
+            )
+
+
+@cache
+def rewriter():
+    from gradio_client import Client
+
+    return Client(REWRITER_SPACE, token=False)
+
+
+def aspect_label(height: int, width: int) -> str:
+    ratio = width / height
+    options = {"21:9": 21 / 9, "16:9": 16 / 9, "4:3": 4 / 3, "1:1": 1.0, "3:4": 3 / 4, "9:16": 9 / 16}
+    return min(options, key=lambda key: abs(options[key] - ratio))
+
+
+def resolve_frames(references, match, duration) -> int:
+    carried = soundtrack_seconds(references)
+    if match and len(carried) == 1 and MIN_FRAMES <= snap_frames(carried[0]) <= MAX_FRAMES:
+        return snap_frames(carried[0])
+    return min(MAX_FRAMES, max(MIN_FRAMES, snap_frames(duration)))
+
+
+def upsample(composer, canvas, match, duration):
+    from gradio_client import handle_file
+
+    composer = dict(composer or {})
+    try:
+        references = references_from(composer)
+        if not any(kind != "audio" for kind, _ in references):
+            raise gr.Error("Add an image or a video reference first.")
+        for kind, limit in REWRITER_LIMITS.items():
+            if sum(k == kind for k, _ in references) > limit:
+                raise gr.Error(
+                    f"The rewriter takes at most {REWRITER_LIMITS['image']} images, 1 video and 1 audio clip."
+                )
+        prompt = (composer.get("upsample_source") or "").strip()
+        if not prompt:
+            raise gr.Error("Describe the video first, then upsample it.")
+        missing = [label for _, label in reference_labels(references) if label not in prompt]
+        if missing:
+            prompt = f"Keep the subject shown in {' and '.join(missing)}. {prompt}"
+        height, width = CANVASES.get(canvas, CANVASES[DEFAULT_CANVAS])
+        num_frames = resolve_frames(references, match, duration)
+        images = [path for kind, path in references if kind == "image"]
+        images += [None] * (REWRITER_LIMITS["image"] - len(images))
+        video = next((path for kind, path in references if kind == "video"), None)
+        audio = next((path for kind, path in references if kind == "audio"), None)
+        rewritten, status = rewriter().predict(
+            task="Ref2AV",
+            prompt=prompt,
+            duration=round(num_frames / FPS, 2),
+            resolution=aspect_label(height, width),
+            image_1=handle_file(images[0]) if images[0] else None,
+            image_2=handle_file(images[1]) if images[1] else None,
+            image_3=handle_file(images[2]) if images[2] else None,
+            image_4=handle_file(images[3]) if images[3] else None,
+            ref_video=handle_file(video) if video else None,
+            ref_audio=handle_file(audio) if audio else None,
+            api_name="/rewrite",
+        )
+        rewritten = (rewritten or "").strip()
+        if not rewritten:
+            raise gr.Error(f"The rewriter returned nothing: {status}")
+    except gr.Error as error:
+        gr.Warning(str(error.message))
+        rewritten = ""
+    except Exception as error:
+        gr.Warning(f"The prompt rewriter failed: {type(error).__name__}: {error}")
+        rewritten = ""
+    composer["upsampled"] = rewritten
+    return composer
+
+
+def _gpu_duration(prompt, references, height, width, num_frames, seed, *args, **kwargs):
     # First call builds the generator (weights load + kernel JIT); later calls
-    # reuse it. 768p denoise is heavier than 480p, so the warm window is larger.
-    return 300 if (_ENGINE is not None) else 900
+    # reuse it. Both scale with the canvas and clip length.
+    base = 300 if (_ENGINE is not None) else 900
+    scale = max(1.0, (height * width * num_frames) / DEFAULT_PIXEL_FRAMES)
+    scale += 0.5 * sum(kind == "video" for kind, _ in references)
+    return int(min(1500, base * scale))
 
 
 @spaces.GPU(duration=_gpu_duration, size="xlarge")
-def generate(subject, scene, soundscape, music, raw_prompt, img1, img2, img3,
-             audio, resolution, length, seed, progress=gr.Progress(track_tqdm=True)):
-    images = [p for p in (_save_image(img1, 1), _save_image(img2, 2), _save_image(img3, 3)) if p]
-    if not images:
-        raise gr.Error("Add at least one reference image.")
-    raw_prompt = (raw_prompt or "").strip()
-    prompt = raw_prompt or build_h3_prompt(subject, scene, soundscape, music, len(images))
-    width, height = RESOLUTIONS.get(resolution, RESOLUTIONS[DEFAULT_RES])
-    num_frames = LENGTHS.get(length, LENGTHS[DEFAULT_LEN])
-    seed = int(seed) if seed is not None else 0
-
+def _generate(prompt, references, height, width, num_frames, seed, progress=gr.Progress(track_tqdm=True)):
     progress(0.02, desc="Loading model (first run only)…")
     eng = get_engine()
     progress(0.1, desc="Generating video + audio…")
-    path = eng.generate(prompt, images, audio, height, width, num_frames, seed)
-    return path
+    return eng.generate(prompt, references, height, width, num_frames, seed)
+
+
+def generate(composer, canvas=DEFAULT_CANVAS, match=True, duration=5, seed=1234):
+    references = references_from(composer)
+    prompt = ((composer or {}).get("prompt") or "").strip()
+    check(prompt, references)
+    height, width = CANVASES.get(canvas, CANVASES[DEFAULT_CANVAS])
+    num_frames = resolve_frames(references, match, duration)
+    seed = int(seed) if seed is not None else 0
+    return _generate(to_h3_prompt(prompt, references), references, height, width, num_frames, seed)
 
 
 # --- UI ---------------------------------------------------------------------
 HEADER = """
 <div class="fh-header">
-  <h1>FastH3 · <span class="fh-accent">OmniRef</span> reference-to-video</h1>
-  <p>FastH3 OmniRef is FastVideo's 8-step reference-to-video student that turns reference images plus a
-  prompt into a short clip with a matching soundtrack. Add one or more reference images (and optional audio),
-  describe the subject and scene, and the prompt is assembled into the model's six-section H3 format for you.</p>
+  <h1>FastH3-<span class="fh-accent">Omni</span></h1>
+  <p>FastH3-Omni is FastVideo's 8-step reference-to-video student that turns reference images, videos and
+  audio plus a prompt into a short clip with a matching soundtrack. Drop in references, then describe the video
+  with the builder or write the prompt yourself, tagging each reference by name.</p>
   <div class="fh-pills">
-    <a href="https://huggingface.co/FastVideo/FastH3-OmniRef-NVFP4-QAD" target="_blank">FastH3 OmniRef</a>
+    <a href="https://huggingface.co/FastVideo/FastVideo-FastH3-Omni-8-Step-V1" target="_blank">Model</a>
+    <a href="https://huggingface.co/collections/FastVideo/fasth3-omni-6ad501acfccdebd3c3b7445b" target="_blank">Collection</a>
     <a href="https://github.com/hao-ai-lab/FastVideo" target="_blank">FastVideo</a>
     <a href="https://huggingface.co/MiniMaxAI/MiniMax-H3" target="_blank">MiniMax-H3</a>
   </div>
 </div>
 """
 
+FASTVIDEO_BLUE = gr.themes.Color(
+    c50="#eef3ff", c100="#dbe5ff", c200="#bccfff", c300="#8eaeff", c400="#5f8cff",
+    c500="#346cff", c600="#1f55ec", c700="#1843c4", c800="#17389c", c900="#16317c", c950="#0e1d4a",
+    name="fastvideo",
+)
+
 CSS = """
-.main.fillable { max-width: 1200px !important; }
+.main.fillable { max-width: 1250px !important; }
 .fh-header { padding: 6px 2px 12px; }
 .fh-header h1 { font-size: 1.6rem; margin: 0 0 6px; font-weight: 700; }
-.fh-header h1 .fh-accent { color: var(--primary-500, #2563eb); }
+.fh-header h1 .fh-accent { color: #346cff; }
 .fh-header p { color: var(--body-text-color-subdued); margin: 0 0 12px; max-width: 76ch; }
 .fh-pills { display: flex; gap: 8px; flex-wrap: wrap; }
 .fh-pills a { font-size: .8rem; padding: 3px 11px; border-radius: 999px;
   background: var(--button-secondary-background-fill); color: var(--body-text-color);
   text-decoration: none; border: 1px solid var(--border-color-primary); }
-#gen-btn { background: linear-gradient(90deg, #2563eb, #1d4ed8); color: #fff; border: none; }
+.fh-pills a:hover { border-color: #346cff; color: #346cff; }
+#gen-btn { background: #346cff; color: #fff; border: none; }
+#gen-btn:hover { background: #1f55ec; }
 """
 
 THEME = gr.themes.Base(
-    primary_hue="blue",
-    secondary_hue="sky",
-    neutral_hue="slate",
+    primary_hue=FASTVIDEO_BLUE,
+    secondary_hue=FASTVIDEO_BLUE,
+    neutral_hue="zinc",
     font=[gr.themes.GoogleFont("Inter"), "ui-sans-serif", "system-ui", "sans-serif"],
+).set(
+    body_background_fill_dark="#000000",
+    block_background_fill_dark="#0b0b0f",
+    background_fill_secondary_dark="#0b0b0f",
+    color_accent="#346cff",
+    slider_color="#346cff",
+    checkbox_background_color_selected="#346cff",
+    checkbox_background_color_selected_dark="#346cff",
 )
 
-with gr.Blocks(title="FastH3 OmniRef", theme=THEME, css=CSS) as demo:
+COMPOSER_HTML = (APP_DIR / "composer.html").read_text()
+COMPOSER_CSS = (APP_DIR / "composer.css").read_text()
+COMPOSER_JS = (APP_DIR / "composer.js").read_text()
+
+with gr.Blocks(title="FastH3-Omni") as demo:
     gr.HTML(HEADER)
 
     with gr.Row():
         with gr.Column(scale=1):
-            with gr.Row():
-                img1 = gr.Image(label="Reference 1 · required", type="numpy", height=190)
-                img2 = gr.Image(label="Reference 2 · optional", type="numpy", height=190)
-                img3 = gr.Image(label="Reference 3 · optional", type="numpy", height=190)
-            with gr.Accordion("Reference audio (optional)", open=False):
-                audio = gr.Audio(label="Voice / sound for the clip to follow", type="filepath")
-            subject = gr.Textbox(
-                label="Subject — what the reference shows",
-                value=DEFAULT_SUBJECT,
-                placeholder="a woman with long hair in a light summer dress",
-            )
-            scene = gr.Textbox(
-                label="Scene — the action, setting and motion",
-                value=DEFAULT_SCENE,
-                lines=3,
-                placeholder="stands in a sunlit meadow and turns to smile at the camera…",
+            composer = gr.HTML(
+                value={"examples": composer_examples()},
+                html_template=COMPOSER_HTML,
+                css_template=COMPOSER_CSS,
+                js_on_load=COMPOSER_JS,
+                apply_default_css=False,
             )
-            run = gr.Button("Generate video", variant="primary", size="lg", elem_id="gen-btn")
+            run = gr.Button("Generate", variant="primary", size="lg", elem_id="gen-btn")
             with gr.Accordion("Advanced options", open=False):
-                soundscape = gr.Textbox(
-                    label="Soundscape — ambient / diegetic sound", value=DEFAULT_SOUND,
+                canvas = gr.Dropdown(label="Canvas", choices=list(CANVASES), value=DEFAULT_CANVAS)
+                match = gr.Checkbox(
+                    label="Match the reference soundtrack",
+                    info="When exactly one reference carries a soundtrack, the clip takes its length.",
+                    value=True,
                 )
-                music = gr.Textbox(label="Non-diegetic music", value=DEFAULT_MUSIC)
-                resolution = gr.Dropdown(
-                    label="Resolution", choices=list(RESOLUTIONS), value=DEFAULT_RES
-                )
-                length = gr.Dropdown(
-                    label="Clip length", choices=list(LENGTHS), value=DEFAULT_LEN
+                duration = gr.Slider(
+                    label="Duration (s)", minimum=MIN_DURATION, maximum=MAX_UI_DURATION, step=1, value=5
                 )
                 seed = gr.Number(label="Seed", value=1234, precision=0)
-                raw_prompt = gr.Textbox(
-                    label="Raw H3 prompt override (optional) — replaces the builder above",
-                    lines=3, value="",
-                )
-                gr.Markdown(
-                    "Prompts are assembled into the six-section H3 format "
-                    "(subject / summary / retention / detailed description / soundscape / music). "
-                    "8 fused steps and guidance 1.0 are fixed by the model's contract."
-                )
+                gr.Markdown("8 fused steps and guidance 1.0 are fixed by the model's contract.")
 
         with gr.Column(scale=1):
-            result = gr.Video(label="Generated clip + soundtrack", height=420, autoplay=True)
-
-    inputs = [subject, scene, soundscape, music, raw_prompt,
-              img1, img2, img3, audio, resolution, length, seed]
-
-    # Populate-only: clicking fills the subject, scene, soundscape, music and the
-    # reference image; the user then presses Generate. Resolution stays at the
-    # recommended 768p default.
-    gr.Examples(
-        label="Examples — click to load a reference + structured prompt, then press Generate",
-        examples=EXAMPLES,
-        inputs=[subject, scene, soundscape, music, img1],
-    )
+            result = gr.Video(label="Video + soundtrack", autoplay=True)
 
-    run.click(generate, inputs=inputs, outputs=[result], api_name="generate")
+    composer.edit(upsample, [composer, canvas, match, duration], [composer], api_name=False, show_progress="hidden")
+    run.click(generate, [composer, canvas, match, duration, seed], [result], api_name="generate")
 
 
 if __name__ == "__main__":
-    demo.queue(max_size=12).launch(server_name="0.0.0.0", server_port=7860, show_error=True)
+    demo.queue(max_size=12).launch(
+        server_name="0.0.0.0",
+        server_port=7860,
+        show_error=True,
+        theme=THEME,
+        css=CSS,
+        ssr_mode=False,
+        allowed_paths=[str(EXAMPLES_DIR)],
+    )

composer.css

@@ -0,0 +1,83 @@
+.fh-composer { --fh-blue: #346cff; --fh-blue-soft: rgba(52, 108, 255, .14); display: flex; flex-direction: column; gap: 14px; }
+.fh-composer [hidden] { display: none !important; }
+.fh-section { border: 1px solid var(--border-color-primary); border-radius: 12px; padding: 12px; background: var(--block-background-fill); }
+.fh-section-head { display: flex; align-items: center; justify-content: space-between; gap: 8px; margin-bottom: 10px; }
+.fh-title { font-weight: 600; font-size: .9rem; }
+.fh-counts { font-size: .75rem; color: var(--body-text-color-subdued); font-variant-numeric: tabular-nums; }
+.fh-tray { display: flex; flex-wrap: wrap; gap: 10px; }
+.fh-drop { width: 132px; height: 150px; border: 1.5px dashed var(--border-color-primary); border-radius: 10px; background: transparent;
+  color: var(--body-text-color-subdued); display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 6px;
+  cursor: pointer; font: inherit; text-align: center; padding: 8px; transition: border-color .15s, background .15s, color .15s; }
+.fh-drop:hover, .fh-tray.is-over .fh-drop { border-color: var(--fh-blue); color: var(--fh-blue); background: var(--fh-blue-soft); }
+.fh-drop-icon { font-size: 1.6rem; line-height: 1; }
+.fh-drop-text { font-size: .78rem; line-height: 1.3; }
+.fh-card { position: relative; width: 132px; height: 150px; border-radius: 10px; overflow: hidden; border: 1px solid var(--border-color-primary);
+  background: var(--background-fill-secondary); cursor: grab; display: flex; flex-direction: column; animation: fh-pop .18s ease-out; }
+.fh-card.is-dragging { opacity: .4; }
+.fh-card.is-target { outline: 2px solid var(--fh-blue); outline-offset: 2px; }
+@keyframes fh-pop { from { transform: scale(.94); opacity: 0; } to { transform: none; opacity: 1; } }
+.fh-thumb { flex: 1; min-height: 0; display: flex; align-items: center; justify-content: center; background: #0b0b0f; }
+.fh-thumb img, .fh-thumb video { width: 100%; height: 100%; object-fit: cover; display: block; }
+.fh-thumb .fh-audio-glyph { color: var(--fh-blue); font-size: 2rem; }
+.fh-card-foot { display: flex; align-items: center; justify-content: space-between; gap: 4px; padding: 6px; }
+.fh-meta { font-size: .68rem; color: var(--body-text-color-subdued); font-variant-numeric: tabular-nums; white-space: nowrap; }
+.fh-x { position: absolute; top: 5px; right: 5px; width: 22px; height: 22px; border-radius: 50%; border: none; background: rgba(0,0,0,.6);
+  color: #fff; cursor: pointer; font-size: 13px; line-height: 22px; padding: 0; opacity: 0; transition: opacity .15s; }
+.fh-card:hover .fh-x, .fh-card:focus-within .fh-x { opacity: 1; }
+.fh-busy { position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; background: rgba(0,0,0,.45); color: #fff; font-size: .75rem; }
+.fh-bad { box-shadow: inset 0 0 0 2px #ef4444; }
+.fh-tag { display: inline-flex; align-items: center; gap: 3px; border: 1px solid transparent; border-radius: 999px; padding: 2px 8px;
+  font-size: .72rem; font-weight: 600; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; cursor: pointer; white-space: nowrap;
+  background: var(--fh-blue-soft); color: var(--fh-blue); transition: background .15s, transform .1s; }
+.fh-tag:hover { background: var(--fh-blue); color: #fff; }
+.fh-tag:active { transform: scale(.96); }
+.fh-tag.is-audio { background: rgba(16, 185, 129, .14); color: #10b981; }
+.fh-tag.is-audio:hover { background: #10b981; color: #fff; }
+.fh-tag.is-video { background: rgba(245, 158, 11, .16); color: #d97706; }
+.fh-tag.is-video:hover { background: #d97706; color: #fff; }
+.fh-tag.is-subject { background: transparent; border-color: var(--fh-blue); }
+.fh-tag.is-missing { outline: 1.5px dashed currentColor; outline-offset: 1px; }
+.fh-hint { font-size: .75rem; color: var(--body-text-color-subdued); margin-top: 8px; min-height: 1em; }
+.fh-hint.is-error { color: #ef4444; }
+.fh-seg { display: inline-flex; padding: 3px; border-radius: 9px; background: var(--background-fill-secondary); border: 1px solid var(--border-color-primary); }
+.fh-seg-btn { border: none; background: transparent; color: var(--body-text-color-subdued); font: inherit; font-size: .82rem; font-weight: 600;
+  padding: 5px 14px; border-radius: 7px; cursor: pointer; }
+.fh-seg-btn.is-active { background: var(--fh-blue); color: #fff; }
+.fh-upsample { display: inline-flex; align-items: center; gap: 6px; border: 1px solid var(--fh-blue); color: var(--fh-blue); background: transparent;
+  border-radius: 9px; padding: 5px 12px; font: inherit; font-size: .82rem; font-weight: 600; cursor: pointer; }
+.fh-upsample:hover { background: var(--fh-blue-soft); }
+.fh-upsample[disabled] { opacity: .6; cursor: progress; }
+.fh-upsample.is-busy .fh-spark { animation: fh-spin 1s linear infinite; display: inline-block; }
+@keyframes fh-spin { to { transform: rotate(360deg); } }
+.fh-pane { display: flex; flex-direction: column; gap: 10px; }
+.fh-field { display: flex; flex-direction: column; gap: 5px; flex: 1; min-width: 0; }
+.fh-label { font-size: .78rem; font-weight: 600; color: var(--body-text-color-subdued); }
+.fh-row { display: flex; gap: 10px; flex-wrap: wrap; }
+.fh-composer textarea, .fh-composer input[type=text] { width: 100%; box-sizing: border-box; border: 1px solid var(--border-color-primary);
+  border-radius: 8px; background: var(--input-background-fill); color: var(--body-text-color); font: inherit; font-size: .9rem; padding: 8px 10px; resize: vertical; }
+.fh-composer textarea:focus, .fh-composer input[type=text]:focus { outline: none; border-color: var(--fh-blue); box-shadow: 0 0 0 3px var(--fh-blue-soft); }
+.fh-composer textarea[data-write] { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: .82rem; line-height: 1.5; }
+.fh-refdescs { display: flex; flex-direction: column; gap: 8px; }
+.fh-refdesc { display: grid; grid-template-columns: auto 1fr; align-items: center; gap: 8px; }
+.fh-refdesc .fh-refdesc-lead { display: flex; flex-direction: column; align-items: flex-start; gap: 3px; min-width: 92px; }
+.fh-refdesc .fh-arrow { font-size: .68rem; color: var(--body-text-color-subdued); }
+.fh-empty { font-size: .82rem; color: var(--body-text-color-subdued); padding: 6px 0; }
+.fh-chipbar { display: flex; flex-wrap: wrap; gap: 5px; align-items: center; min-height: 0; }
+.fh-chipbar:empty { display: none; }
+.fh-chipbar .fh-chipbar-label { font-size: .7rem; color: var(--body-text-color-subdued); margin-right: 2px; }
+.fh-preview summary { cursor: pointer; font-size: .78rem; font-weight: 600; color: var(--body-text-color-subdued); display: flex; align-items: center; gap: 10px; }
+.fh-preview pre { white-space: pre-wrap; font-size: .74rem; line-height: 1.45; max-height: 260px; overflow: auto; background: var(--background-fill-secondary);
+  border-radius: 8px; padding: 10px; margin: 8px 0 0; }
+.fh-link { border: none; background: none; color: var(--fh-blue); font: inherit; font-size: .78rem; cursor: pointer; padding: 0; }
+.fh-lint { font-size: .75rem; display: flex; flex-wrap: wrap; gap: 6px; align-items: center; color: var(--body-text-color-subdued); }
+.fh-lint .is-ok { color: #10b981; }
+.fh-lint .is-warn { color: #d97706; }
+.fh-lint .is-err { color: #ef4444; }
+.fh-examples { display: flex; flex-direction: column; gap: 8px; }
+.fh-example-row { display: flex; gap: 10px; overflow-x: auto; padding-bottom: 2px; }
+.fh-example { display: flex; align-items: center; gap: 8px; border: 1px solid var(--border-color-primary); border-radius: 10px; padding: 6px 10px 6px 6px;
+  background: transparent; color: var(--body-text-color); cursor: pointer; font: inherit; font-size: .8rem; text-align: left; min-width: 0; flex: 1 1 0; }
+.fh-example:hover { border-color: var(--fh-blue); background: var(--fh-blue-soft); }
+.fh-example img { width: 44px; height: 44px; border-radius: 7px; object-fit: cover; flex: none; }
+.fh-example span { overflow: hidden; text-overflow: ellipsis; display: -webkit-box; -webkit-line-clamp: 2; -webkit-box-orient: vertical; }
+.fh-hidden { display: none !important; }

composer.html

@@ -0,0 +1,66 @@
+<div id="fh-composer" class="fh-composer">
+  <section class="fh-section">
+    <div class="fh-section-head">
+      <span class="fh-title">References</span>
+      <span class="fh-counts" data-counts></span>
+    </div>
+    <div class="fh-tray" data-tray>
+      <button type="button" class="fh-drop" data-drop>
+        <span class="fh-drop-icon">+</span>
+        <span class="fh-drop-text">Drop images, videos or audio<br><small>or click to add · order matters</small></span>
+      </button>
+    </div>
+    <input type="file" data-file multiple accept="image/*,video/*,audio/*" hidden>
+    <div class="fh-hint" data-ref-hint></div>
+  </section>
+
+  <section class="fh-section">
+    <div class="fh-section-head">
+      <div class="fh-seg" role="tablist">
+        <button type="button" class="fh-seg-btn is-active" data-mode="builder">Builder</button>
+        <button type="button" class="fh-seg-btn" data-mode="write">Write</button>
+      </div>
+      <button type="button" class="fh-upsample" data-upsample title="Rewrite into a detailed H3 prompt with the MiniMax-H3 rewriter">
+        <span class="fh-spark">✦</span><span data-upsample-label>Upsample</span>
+      </button>
+    </div>
+
+    <div class="fh-pane" data-pane="builder">
+      <div class="fh-refdescs" data-refdescs></div>
+      <label class="fh-field">
+        <span class="fh-label">Action &amp; scene</span>
+        <div class="fh-chipbar" data-subject-chips></div>
+        <textarea data-b="scene" rows="3" placeholder="<Subject 1> walks along a sunlit beach promenade at golden hour, glancing toward the camera…"></textarea>
+      </label>
+      <div class="fh-row">
+        <label class="fh-field">
+          <span class="fh-label">Soundscape</span>
+          <input data-b="sound" type="text" placeholder="ocean waves, a gentle breeze and distant seagulls">
+        </label>
+        <label class="fh-field">
+          <span class="fh-label">Music</span>
+          <input data-b="music" type="text" placeholder="none">
+        </label>
+      </div>
+      <label class="fh-field">
+        <span class="fh-label">Style</span>
+        <input data-b="style" type="text" value="realistic photographic">
+      </label>
+      <details class="fh-preview">
+        <summary>H3 prompt preview <button type="button" class="fh-link" data-to-write>Edit as text →</button></summary>
+        <pre data-preview></pre>
+      </details>
+    </div>
+
+    <div class="fh-pane" data-pane="write" hidden>
+      <div class="fh-chipbar" data-write-chips></div>
+      <textarea data-write rows="9" placeholder="Describe the video and refer to each reference by its tag, e.g. <Picture 1> dances to <Audio 1> in a neon-lit club…"></textarea>
+      <div class="fh-lint" data-lint></div>
+    </div>
+  </section>
+
+  <section class="fh-section fh-examples">
+    <span class="fh-title">Try an example</span>
+    <div class="fh-example-row" data-examples></div>
+  </section>
+</div>

composer.js

@@ -0,0 +1,554 @@
+const LIMITS = { image: 9, video: 3, audio: 3 };
+const NAMES = { image: "Picture", video: "Video", audio: "Audio" };
+const PLACEHOLDERS = {
+  image: "who or what is this? e.g. a woman with a curly afro in a white t-shirt",
+  video: "what to take from it? e.g. the dance moves and camera push",
+  audio: "what is it? e.g. her voice, or upbeat synth-pop",
+};
+const VIDEO_MIN = 2, VIDEO_MAX = 15;
+const $ = (sel) => element.querySelector(sel);
+
+const initial = props.value || {};
+const state = {
+  mode: initial.mode || "builder",
+  refs: (initial.refs || []).map((ref) => ({ ...ref, id: ref.id || crypto.randomUUID() })),
+  builder: Object.assign({ scene: "", sound: "", music: "", style: "realistic photographic" }, initial.builder || {}),
+  write: initial.write || "",
+  writeEdited: !!initial.writeEdited,
+};
+let lastPushed = "";
+let upsampling = false;
+
+function kindOf(file) {
+  const type = file.type || "";
+  if (type.startsWith("image/")) return "image";
+  if (type.startsWith("video/")) return "video";
+  if (type.startsWith("audio/")) return "audio";
+  const ext = (file.name.split(".").pop() || "").toLowerCase();
+  if (["png", "jpg", "jpeg", "webp", "gif", "bmp"].includes(ext)) return "image";
+  if (["mp4", "mov", "webm", "mkv", "avi"].includes(ext)) return "video";
+  if (["wav", "mp3", "flac", "ogg", "m4a", "aac", "opus"].includes(ext)) return "audio";
+  return null;
+}
+
+function labelFor(index) {
+  const ref = state.refs[index];
+  let n = 0;
+  for (let i = 0; i <= index; i++) if (state.refs[i].kind === ref.kind) n++;
+  return `<${NAMES[ref.kind]} ${n}>`;
+}
+
+function labels() {
+  return state.refs.map((ref, index) => ({ ref, index, label: labelFor(index) }));
+}
+
+function subjects() {
+  return labels().filter(({ ref }) => ref.kind !== "audio");
+}
+
+function count(kind) {
+  return state.refs.filter((ref) => ref.kind === kind).length;
+}
+
+function escapeHtml(text) {
+  return String(text).replace(/[&<>"']/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[c]);
+}
+
+function clean(text) {
+  return (text || "").trim().replace(/[.\s]+$/, "");
+}
+
+function buildPrompt() {
+  const b = state.builder;
+  const all = labels();
+  const visual = subjects();
+  const audio = all.filter(({ ref }) => ref.kind === "audio");
+  const defs = [];
+  const retention = [];
+  visual.forEach(({ ref, label }, k) => {
+    const subject = `<Subject ${k + 1}>`;
+    const what = clean(ref.desc) || (ref.kind === "video" ? "the subject and motion of the reference video" : "the reference subject");
+    defs.push(`${subject} is ${what}, shown in ${label}.`);
+    retention.push(`${subject}: fully_preserved - the identity, appearance and details from ${label} are kept.`);
+  });
+  audio.forEach(({ ref, label }) => {
+    defs.push(`${label} is ${clean(ref.desc) || "the voice and sound reference"}.`);
+  });
+  const names = visual.map(({ ref }, k) => clean(ref.desc) || `<Subject ${k + 1}>`);
+  const summaryWho = names.length ? names.join(" and ") : "the scene";
+  let scene = (b.scene || "").trim();
+  if (scene && visual.length && !/<Subject \d+>/.test(scene) && /^[a-z]/.test(scene)) scene = `<Subject 1> ${scene}`;
+  const style = clean(b.style) || "realistic photographic";
+  let sound = clean(b.sound) || "natural ambient sound that matches the scene";
+  const unmentioned = audio.filter(({ label }) => !sound.includes(label)).map(({ label }) => label);
+  if (unmentioned.length) sound += `, following ${unmentioned.join(" and ")}`;
+  return [
+    `subject_definitions:\n${defs.join("\n") || "<Subject 1> is the subject of the video."}`,
+    `summary:\nA short video in a ${style} style of ${summaryWho}.`,
+    `retention_analysis:\n${retention.join("\n") || "none"}`,
+    `detailed_description:\nThe target video is in a ${style} style.\n[Shot 1] ${scene || "The subject moves naturally in the scene."}`,
+    `overall_soundscape:\n${sound}.`,
+    `non_diegetic_music:\n${clean(b.music) || "none"}.`,
+  ].join("\n\n");
+}
+
+function upsampleSource() {
+  if (state.mode === "write") return state.write;
+  const parts = [];
+  subjects().forEach(({ ref, label }, k) => parts.push(`${label} shows ${clean(ref.desc) || `subject ${k + 1}`}.`));
+  labels().filter(({ ref }) => ref.kind === "audio").forEach(({ ref, label }) => parts.push(`${label} is ${clean(ref.desc) || "the voice and sound reference"}.`));
+  const b = state.builder;
+  let scene = (b.scene || "").replace(/<Subject (\d+)>/g, (_, n) => (subjects()[n - 1] ? clean(subjects()[n - 1].ref.desc) || `subject ${n}` : `subject ${n}`));
+  if (scene) parts.push(scene.trim());
+  if (clean(b.sound)) parts.push(`Sound: ${clean(b.sound)}.`);
+  if (clean(b.music) && clean(b.music).toLowerCase() !== "none") parts.push(`Music: ${clean(b.music)}.`);
+  if (clean(b.style) && clean(b.style) !== "realistic photographic") parts.push(`Style: ${clean(b.style)}.`);
+  return parts.join(" ");
+}
+
+function finalPrompt() {
+  return state.mode === "write" ? state.write : buildPrompt();
+}
+
+function push() {
+  const value = {
+    mode: state.mode,
+    refs: state.refs.map(({ id, kind, path, url, name, desc, duration, uploading, error }) => ({ id, kind, path, url, name, desc, duration, uploading: !!uploading, error: error || null })),
+    builder: { ...state.builder },
+    write: state.write,
+    writeEdited: state.writeEdited,
+    prompt: finalPrompt(),
+    upsample_source: upsampleSource(),
+  };
+  lastPushed = JSON.stringify(value);
+  props.value = value;
+}
+
+function insertAt(target, text) {
+  const start = target.selectionStart ?? target.value.length;
+  const end = target.selectionEnd ?? target.value.length;
+  const before = target.value.slice(0, start);
+  const pad = before && !/\s$/.test(before) ? " " : "";
+  target.value = before + pad + text + " " + target.value.slice(end);
+  const caret = start + pad.length + text.length + 1;
+  target.focus();
+  target.setSelectionRange(caret, caret);
+  target.dispatchEvent(new Event("input", { bubbles: true }));
+}
+
+function tagButton(label, kind, onClick, extra = "") {
+  const button = document.createElement("button");
+  button.type = "button";
+  button.className = `fh-tag ${kind === "audio" ? "is-audio" : kind === "video" ? "is-video" : kind === "subject" ? "is-subject" : ""} ${extra}`;
+  button.textContent = label;
+  button.title = "Insert into the prompt";
+  button.addEventListener("click", (event) => { event.preventDefault(); event.stopPropagation(); onClick(); });
+  return button;
+}
+
+function insertLabel(label) {
+  if (state.mode === "write") {
+    insertAt($("[data-write]"), label);
+  } else {
+    insertAt($('[data-b="scene"]'), label);
+  }
+}
+
+function fmtSeconds(seconds) {
+  return seconds == null ? "" : `${seconds.toFixed(1)}s`;
+}
+
+function renderTray() {
+  const tray = $("[data-tray]");
+  tray.querySelectorAll(".fh-card").forEach((card) => card.remove());
+  const drop = $("[data-drop]");
+  labels().forEach(({ ref, index, label }) => {
+    const card = document.createElement("div");
+    card.className = "fh-card" + (ref.error ? " fh-bad" : "");
+    card.draggable = true;
+    card.dataset.index = index;
+    const thumb = document.createElement("div");
+    thumb.className = "fh-thumb";
+    if (ref.kind === "image") {
+      thumb.innerHTML = `<img alt="" src="${escapeHtml(ref.preview || ref.url || "")}">`;
+    } else if (ref.kind === "video") {
+      thumb.innerHTML = `<video muted playsinline preload="metadata" src="${escapeHtml((ref.preview || ref.url || "") + "#t=0.1")}"></video>`;
+      const video = thumb.querySelector("video");
+      card.addEventListener("mouseenter", () => video.play().catch(() => {}));
+      card.addEventListener("mouseleave", () => { video.pause(); });
+    } else {
+      thumb.innerHTML = `<span class="fh-audio-glyph">♫</span>`;
+      card.addEventListener("click", (event) => {
+        if (event.target.closest("button")) return;
+        if (!ref._audio) ref._audio = new Audio(ref.preview || ref.url);
+        ref._audio.paused ? ref._audio.play() : ref._audio.pause();
+      });
+      card.title = "Click to play";
+    }
+    card.appendChild(thumb);
+    const foot = document.createElement("div");
+    foot.className = "fh-card-foot";
+    foot.appendChild(tagButton(label, ref.kind, () => insertLabel(label)));
+    const meta = document.createElement("span");
+    meta.className = "fh-meta";
+    meta.textContent = ref.error ? "!" : fmtSeconds(ref.duration);
+    foot.appendChild(meta);
+    card.appendChild(foot);
+    const remove = document.createElement("button");
+    remove.type = "button";
+    remove.className = "fh-x";
+    remove.textContent = "×";
+    remove.title = "Remove";
+    remove.addEventListener("click", (event) => { event.stopPropagation(); removeRef(index); });
+    card.appendChild(remove);
+    if (ref.uploading) {
+      const busy = document.createElement("div");
+      busy.className = "fh-busy";
+      busy.textContent = "uploading…";
+      card.appendChild(busy);
+    }
+    card.addEventListener("dragstart", (event) => {
+      card.classList.add("is-dragging");
+      event.dataTransfer.setData("text/fh-index", String(index));
+      event.dataTransfer.effectAllowed = "move";
+    });
+    card.addEventListener("dragend", () => card.classList.remove("is-dragging"));
+    card.addEventListener("dragover", (event) => {
+      if (!event.dataTransfer.types.includes("text/fh-index")) return;
+      event.preventDefault();
+      card.classList.add("is-target");
+    });
+    card.addEventListener("dragleave", () => card.classList.remove("is-target"));
+    card.addEventListener("drop", (event) => {
+      const from = event.dataTransfer.getData("text/fh-index");
+      card.classList.remove("is-target");
+      if (from === "") return;
+      event.preventDefault();
+      event.stopPropagation();
+      moveRef(Number(from), index);
+    });
+    tray.insertBefore(card, drop);
+  });
+  drop.hidden = state.refs.length >= LIMITS.image + LIMITS.video + LIMITS.audio;
+  $("[data-counts]").textContent = `${count("image")}/${LIMITS.image} images · ${count("video")}/${LIMITS.video} videos · ${count("audio")}/${LIMITS.audio} audio`;
+  const hint = $("[data-ref-hint]");
+  const bad = state.refs.find((ref) => ref.error);
+  if (bad) {
+    hint.textContent = bad.error;
+    hint.className = "fh-hint is-error";
+  } else if (!state.refs.length) {
+    hint.textContent = "Add at least one image or video. Each reference gets a tag you can use in the prompt.";
+    hint.className = "fh-hint";
+  } else if (!state.refs.some((ref) => ref.kind !== "audio")) {
+    hint.textContent = "Audio needs an image or a video alongside it.";
+    hint.className = "fh-hint is-error";
+  } else {
+    hint.textContent = "Drag to reorder: the order numbers the tags. Click a tag to insert it into the prompt.";
+    hint.className = "fh-hint";
+  }
+}
+
+function renderBuilder() {
+  const host = $("[data-refdescs]");
+  host.innerHTML = "";
+  if (!state.refs.length) {
+    host.innerHTML = `<div class="fh-empty">Add references above, then describe what each one is.</div>`;
+  }
+  let subjectIndex = 0;
+  labels().forEach(({ ref, label }) => {
+    const row = document.createElement("div");
+    row.className = "fh-refdesc";
+    const lead = document.createElement("div");
+    lead.className = "fh-refdesc-lead";
+    lead.appendChild(tagButton(label, ref.kind, () => insertLabel(label)));
+    if (ref.kind !== "audio") {
+      subjectIndex++;
+      const arrow = document.createElement("span");
+      arrow.className = "fh-arrow";
+      arrow.textContent = `→ <Subject ${subjectIndex}>`;
+      lead.appendChild(arrow);
+    }
+    const input = document.createElement("input");
+    input.type = "text";
+    input.placeholder = PLACEHOLDERS[ref.kind];
+    input.value = ref.desc || "";
+    input.addEventListener("input", () => { ref.desc = input.value; renderSubjectChips(); renderPreview(); push(); });
+    row.appendChild(lead);
+    row.appendChild(input);
+    host.appendChild(row);
+  });
+  renderSubjectChips();
+  renderPreview();
+}
+
+function renderSubjectChips() {
+  const bar = $("[data-subject-chips]");
+  bar.innerHTML = "";
+  const visual = subjects();
+  if (!visual.length) return;
+  const lead = document.createElement("span");
+  lead.className = "fh-chipbar-label";
+  lead.textContent = "Insert:";
+  bar.appendChild(lead);
+  visual.forEach(({ ref }, k) => {
+    const label = `<Subject ${k + 1}>`;
+    const chip = tagButton(label, "subject", () => insertAt($('[data-b="scene"]'), label));
+    if (clean(ref.desc)) chip.title = clean(ref.desc);
+    bar.appendChild(chip);
+  });
+  labels().filter(({ ref }) => ref.kind === "audio").forEach(({ label }) => {
+    bar.appendChild(tagButton(label, "audio", () => insertAt($('[data-b="sound"]'), label)));
+  });
+}
+
+function renderPreview() {
+  $("[data-preview]").textContent = buildPrompt();
+}
+
+function renderWrite() {
+  const bar = $("[data-write-chips]");
+  bar.innerHTML = "";
+  if (state.refs.length) {
+    const lead = document.createElement("span");
+    lead.className = "fh-chipbar-label";
+    lead.textContent = "Insert:";
+    bar.appendChild(lead);
+  }
+  const text = state.write || "";
+  labels().forEach(({ ref, label }) => {
+    bar.appendChild(tagButton(label, ref.kind, () => insertAt($("[data-write]"), label), text.includes(label) ? "" : "is-missing"));
+  });
+  const textarea = $("[data-write]");
+  if (textarea.value !== text) textarea.value = text;
+  renderLint();
+}
+
+function renderLint() {
+  const lint = $("[data-lint]");
+  const text = state.write || "";
+  const known = new Set(labels().map(({ label }) => label));
+  const used = [...new Set(text.match(/<(Picture|Video|Audio) \d+>/g) || [])];
+  const unknown = used.filter((label) => !known.has(label));
+  const missing = [...known].filter((label) => !text.includes(label));
+  const structured = /subject_definitions:/.test(text);
+  const parts = [];
+  if (text.includes("<removed>")) parts.push(`<span class="is-err">The prompt still refers to a removed reference (&lt;removed&gt;)</span>`);
+  if (!text.trim()) parts.push(`<span>Write a prompt, or switch to Builder.</span>`);
+  if (unknown.length) parts.push(`<span class="is-err">No reference for ${escapeHtml(unknown.join(", "))}</span>`);
+  if (missing.length && text.trim()) parts.push(`<span class="is-warn">Not mentioned: ${escapeHtml(missing.join(", "))}</span>`);
+  if (text.trim() && !unknown.length && !missing.length && !text.includes("<removed>")) parts.push(`<span class="is-ok">✓ every reference is mentioned</span>`);
+  if (text.trim()) parts.push(`<span>${structured ? "H3 six-section prompt, used as is" : "Plain prompt, wrapped into the H3 format on generate"}</span>`);
+  lint.innerHTML = parts.join("");
+}
+
+function renderMode() {
+  element.querySelectorAll("[data-mode]").forEach((button) => button.classList.toggle("is-active", button.dataset.mode === state.mode));
+  $('[data-pane="builder"]').hidden = state.mode !== "builder";
+  $('[data-pane="write"]').hidden = state.mode !== "write";
+}
+
+function renderBuilderFields() {
+  element.querySelectorAll("[data-b]").forEach((input) => {
+    const key = input.dataset.b;
+    if (input.value !== (state.builder[key] || "")) input.value = state.builder[key] || "";
+  });
+}
+
+function renderAll() {
+  renderMode();
+  renderTray();
+  renderBuilderFields();
+  renderBuilder();
+  renderWrite();
+}
+
+function relabelText(before) {
+  if (!state.write) return;
+  if (!state.writeEdited) {
+    state.write = buildPrompt();
+    return;
+  }
+  const after = labels();
+  const map = {};
+  before.forEach(({ id, label }) => {
+    const now = after.find(({ ref }) => ref.id === id);
+    map[label] = now ? now.label : null;
+  });
+  state.write = state.write.replace(/<(Picture|Video|Audio) \d+>/g, (label) => (label in map ? (map[label] === null ? "<removed>" : `\u0000${map[label]}\u0000`) : label)).replace(/\u0000/g, "");
+}
+
+function snapshotLabels() {
+  return labels().map(({ ref, label }) => ({ id: ref.id, label }));
+}
+
+function removeRef(index) {
+  const before = snapshotLabels();
+  const [ref] = state.refs.splice(index, 1);
+  if (ref && ref._audio) ref._audio.pause();
+  relabelText(before);
+  renderAll();
+  push();
+}
+
+function moveRef(from, to) {
+  if (from === to) return;
+  const before = snapshotLabels();
+  const [ref] = state.refs.splice(from, 1);
+  state.refs.splice(to, 0, ref);
+  relabelText(before);
+  renderAll();
+  push();
+}
+
+function mediaDuration(url, kind) {
+  return new Promise((resolve) => {
+    const media = document.createElement(kind === "video" ? "video" : "audio");
+    media.preload = "metadata";
+    media.onloadedmetadata = () => resolve(Number.isFinite(media.duration) ? media.duration : null);
+    media.onerror = () => resolve(null);
+    media.src = url;
+  });
+}
+
+async function addFiles(files) {
+  for (const file of files) {
+    const kind = kindOf(file);
+    if (!kind) continue;
+    if (count(kind) >= LIMITS[kind]) {
+      const hint = $("[data-ref-hint]");
+      hint.textContent = `At most ${LIMITS[kind]} ${kind} references.`;
+      hint.className = "fh-hint is-error";
+      continue;
+    }
+    const ref = { id: crypto.randomUUID(), kind, name: file.name, desc: "", preview: URL.createObjectURL(file), uploading: true };
+    state.refs.push(ref);
+    if (kind !== "image") {
+      ref.duration = await mediaDuration(ref.preview, kind);
+      if (kind === "video" && ref.duration != null && (ref.duration < VIDEO_MIN || ref.duration > VIDEO_MAX)) {
+        ref.error = `${file.name} is ${ref.duration.toFixed(1)} s; reference videos must be ${VIDEO_MIN}–${VIDEO_MAX} s.`;
+      }
+    }
+    renderAll();
+    push();
+    upload(file).then(({ path, url }) => {
+      ref.path = path;
+      ref.url = url;
+      ref.uploading = false;
+      renderTray();
+      push();
+    }).catch((error) => {
+      ref.uploading = false;
+      ref.error = `Upload failed for ${file.name}: ${error}`;
+      renderTray();
+      push();
+    });
+  }
+}
+
+const fileInput = $("[data-file]");
+$("[data-drop]").addEventListener("click", () => fileInput.click());
+fileInput.addEventListener("change", () => { addFiles([...fileInput.files]); fileInput.value = ""; });
+const tray = $("[data-tray]");
+tray.addEventListener("dragover", (event) => {
+  if (!event.dataTransfer.types.includes("Files")) return;
+  event.preventDefault();
+  tray.classList.add("is-over");
+});
+tray.addEventListener("dragleave", (event) => { if (!tray.contains(event.relatedTarget)) tray.classList.remove("is-over"); });
+tray.addEventListener("drop", (event) => {
+  tray.classList.remove("is-over");
+  if (!event.dataTransfer.files.length) return;
+  event.preventDefault();
+  addFiles([...event.dataTransfer.files]);
+});
+
+element.querySelectorAll("[data-mode]").forEach((button) => button.addEventListener("click", () => {
+  if (button.dataset.mode === state.mode) return;
+  if (button.dataset.mode === "write" && (!state.write.trim() || !state.writeEdited)) {
+    state.write = buildPrompt();
+    state.writeEdited = false;
+  }
+  state.mode = button.dataset.mode;
+  renderAll();
+  push();
+}));
+
+$("[data-to-write]").addEventListener("click", (event) => {
+  event.preventDefault();
+  state.write = buildPrompt();
+  state.writeEdited = false;
+  state.mode = "write";
+  renderAll();
+  push();
+});
+
+element.querySelectorAll("[data-b]").forEach((input) => input.addEventListener("input", () => {
+  state.builder[input.dataset.b] = input.value;
+  renderPreview();
+  push();
+}));
+
+$("[data-write]").addEventListener("input", (event) => {
+  state.write = event.target.value;
+  state.writeEdited = true;
+  renderWrite();
+  push();
+});
+
+const upsampleButton = $("[data-upsample]");
+function setUpsampling(busy) {
+  upsampling = busy;
+  upsampleButton.disabled = busy;
+  upsampleButton.classList.toggle("is-busy", busy);
+  $("[data-upsample-label]").textContent = busy ? "Upsampling…" : "Upsample";
+}
+upsampleButton.addEventListener("click", () => {
+  if (upsampling) return;
+  setUpsampling(true);
+  push();
+  trigger("edit");
+});
+
+function loadExample(example) {
+  state.refs = example.refs.map((ref) => ({ ...ref, id: crypto.randomUUID(), preview: ref.url }));
+  state.builder = Object.assign({ scene: "", sound: "", music: "", style: "realistic photographic" }, example.builder);
+  state.write = "";
+  state.writeEdited = false;
+  state.mode = "builder";
+  renderAll();
+  push();
+}
+
+const examples = (props.value && props.value.examples) || [];
+const exampleRow = $("[data-examples]");
+examples.forEach((example) => {
+  const button = document.createElement("button");
+  button.type = "button";
+  button.className = "fh-example";
+  button.innerHTML = `<img alt="" src="${escapeHtml(example.refs[0].url)}"><span>${escapeHtml(example.title)}</span>`;
+  button.addEventListener("click", () => loadExample(example));
+  exampleRow.appendChild(button);
+});
+if (!examples.length) $(".fh-examples").hidden = true;
+
+watch("value", () => {
+  const incoming = props.value || {};
+  if (JSON.stringify(incoming) === lastPushed) return;
+  if ("upsampled" in incoming) {
+    setUpsampling(false);
+    if (incoming.upsampled) {
+      state.write = incoming.upsampled;
+      state.writeEdited = true;
+      state.mode = "write";
+    }
+    renderAll();
+    push();
+  }
+});
+
+if (!state.refs.length && examples.length) {
+  loadExample(examples[0]);
+} else {
+  renderAll();
+  push();
+}

fasth3_engine.py

@@ -120,15 +120,13 @@ class FastH3OmniRefEngine:
         self.generator = VideoGenerator.from_config(cfg)
         print(f"[fasth3-fp8] generator ready in {time.time() - t0:.1f}s", flush=True)
 
-    def generate(self, prompt, image_paths, audio_path, height, width, num_frames, seed) -> str:
+    def generate(self, prompt, references, height, width, num_frames, seed) -> str:
         if self.generator is None:
             raise RuntimeError("engine not initialized")
         from fastvideo.api import GenerationRequest, InputConfig, OutputConfig, SamplingConfig
         from fastvideo.pipelines.basic.minimax_h3 import MiniMaxH3Reference
 
-        references = [MiniMaxH3Reference(source=p, media_type="image") for p in image_paths]
-        if audio_path:
-            references.append(MiniMaxH3Reference(source=audio_path, media_type="audio"))
+        references = [MiniMaxH3Reference(source=path, media_type=kind) for kind, path in references]
 
         out_dir = tempfile.mkdtemp(prefix="fasth3_fp8_out_")
         out_path = os.path.join(out_dir, "fasth3_omniref.mp4")

requirements.txt

@@ -1,8 +1,8 @@
-# FastVideo main @ the FastH3 OmniRef PDD + NVFP4 consumer-profile commit, with
-# the in-process UniprocExecutor. Pulls torch==2.12.0, gradio==5.32.0,
-# transformers>=5.15.0. cu130 wheels for Blackwell sm_120 (RTX PRO 6000).
+# FastVideo main @ 376e445f (the FastH3 OmniRef PDD + NVFP4 consumer-profile commit),
+# rebuilt with its unused gradio==5.32.0 pin relaxed so the Space runs Gradio 6.
+# Pulls torch==2.12.0, transformers>=5.15.0. cu130 wheels for Blackwell sm_120.
 --extra-index-url https://download.pytorch.org/whl/cu130
-fastvideo @ git+https://github.com/hao-ai-lab/FastVideo.git@376e445f0e8d192c2d9a5f28010a6b3a7012667e
+fastvideo @ https://huggingface.co/datasets/multimodalart/zerogpu-blackwell-wheels/resolve/main/wheels/fastvideo/376e445-gradio-unpinned/fastvideo-0.2.1-py3-none-any.whl
 
 # NVFP4 FP4 GEMMs (packed linears + decoder) on Blackwell.
 flashinfer-python