Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+text_tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/example_first_frame.png filter=lfs diff=lfs merge=lfs -text
+assets/example_output.mp4 filter=lfs diff=lfs merge=lfs -text
+images/aa-i2v-all-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text
+images/aa-i2v-open-models-2026-07-17.png filter=lfs diff=lfs merge=lfs -text

BIAS.md

@@ -0,0 +1,11 @@
+## Bias
+
+| Field | Response |
+| :---- | :---- |
+| Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing | None. |
+| Measures taken to mitigate against unwanted bias | Training, evaluation, and testing data are curated before release to filter restricted content, including content relating to protected classes. Model behavior is evaluated across Physical AI domains — robotics, autonomous vehicles, human-centric scenes, common scenes, industry, miscellaneous, and physics-oriented benchmarks — with attention to coverage across diverse demographic and contextual characteristics that affect protected-class outcomes. |
+| Which characteristic (feature) show(s) the greatest difference in performance?: | Greatest performance differences are observed in tasks requiring long-horizon temporal consistency, fine-grained physical interactions, and embodiment-specific action generation. Performance is generally stronger on common visual reasoning and world-generation tasks than on complex multi-agent, robotics-control, or tightly synchronized multimodal generation scenarios. |
+| Which feature(s) have the worst performance overall? | Performance is generally weakest in tasks requiring long-horizon temporal consistency, precise physical interactions, embodiment-specific action control, and strict audio-visual synchronization. |
+| If using internal data, description of methods implemented in data acquisition or processing, if any, to address the prevalence of identifiable biases in the training, testing, and validation data: | Bias-specific methods applied during data processing include person-presence screening, demographic-taxonomy classification (age, gender, ethnicity), embedding-based diversity analysis, and dataset balancing across sources. Internal analysis surfaced: non-person scenes are more prevalent than person-centric content; demographic-taxonomy outputs on person-present samples are most frequently "uncertain" across age, gender, and ethnicity dimensions; and source-type variation, with people-centric image and video datasets showing higher demographic signal than document-, object-, robotics-, or scene-focused datasets. *(Quantitative details in the row below.)* Downstream deployments should add bias audits, fairness evaluation, red-teaming, demographically balanced fine-tuning, or counterfactual augmentation as mitigations. |
+| Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | Dataset analytics pipelines, metadata distribution analysis, heuristic quality checks, embedding-based clustering, model-assisted filtering systems, and benchmark evaluation suites are used to assess statistical imbalances and identify patterns that may introduce bias into model behavior. |
+| Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | These datasets, such as OpenImages-derived detection-to-NLP datasets, visual grounding and VQA datasets, document/image understanding datasets, video/action understanding datasets, and NVIDIA-created or curated visual datasets, do not collectively or exhaustively represent all demographic groups (and proportionally therein). For instance, automated person-presence screening did not identify a person in approximately 58% of visual samples analyzed across approximately 400 datasets, while person-present signals were identified in approximately 42% of analyzed samples. In the subset where person-present signals were identified, these datasets contain uneven representation splits across the measured visual taxonomies: age outputs were most frequently uncertain, followed by child and adult; gender outputs were most frequently uncertain, followed by male and female; and ethnicity outputs were most frequently uncertain, followed by Hispanic and White as the most frequent identified categories. Dataset-level results vary by source type, with people-centric image and video datasets containing higher person-present and demographic-taxonomy signals than document-, object-, robotics-, or scene-focused datasets. To mitigate these imbalances, we recommend considering evaluation techniques such as bias audits, task-specific fairness evaluation, and red-teaming, along with fine-tuning with demographically balanced datasets and counterfactual data augmentation to align with the desired model behavior. This evaluation used a baseline of 200 samples across all datasets, with larger subsets of up to 3,000 samples utilized for certain in-depth analyses, identified as optimal thresholds for maximizing embedder accuracy. |

EXPLAINABILITY.md

@@ -0,0 +1,16 @@
+## Explainability
+
+| Field | Response |
+| :---- | :---- |
+| Intended Application & Domain | World reasoning and generation for Physical AI. |
+| Model Type | Mixture-of-Transformers architecture with two towers. One is an autoregressive model for Physical AI reasoning; the other is a diffusion model for Physical AI generation. |
+| Intended Users | Physical AI developers, researchers, and practitioners building or evaluating autonomous vehicle, robotics, and world-generation workflows. |
+| Output | Images, videos, audio, and action commands. |
+| Tools used to evaluate datasets to identify synthetic data and ensure data authenticity. | Dataset provenance analysis, metadata validation, watermark and artifact detection, embedding-based clustering, heuristic quality checks, and model-assisted data validation pipelines are used to identify synthetic content patterns, assess dataset authenticity, and improve data quality during dataset curation. |
+| Describe how the model works | Cosmos3 is an Omni world foundation model that generates texts, images, videos, audio, and action commands from combinations of text, images, videos, and action trajectory inputs. Input tokens from multiple modalities are packed into a shared sequence and processed by our mixture-of-transformer backbone with modality-specific output heads. |
+| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | None. |
+| Technical Limitations | The model may not follow text, image, video, audio, or action trajectory inputs accurately in challenging cases, especially where the input contains complex scene composition, unusual camera motion, multiple interacting agents, low lighting, high motion blur, or fine-grained physical interactions. Generated outputs may contain temporal inconsistency, object morphing, inaccurate 3D structure, or implausible physical dynamics. Generated audio may not accurately render intelligible speech, or maintain strict temporal and semantic alignment with the visual context. |
+| Verified to have met prescribed NVIDIA quality standards | Yes. |
+| Performance Metrics | Video generation is measured using PAIBench-G, RBench, PhysicsIQ, and Artifical Analysis Image2Video benchmark. Image generation uses UniGenBench and Artifical Analysis Text2Image benchmark. For transfer evaluation, we use PAIBench-C and AVBench-C. Audio generation uses internal benchmarks. Action prediction uses metrics such as action MSE, Absolute Translation Error, Relative Translation Error, Relative Rotation Error, PSNR, and robotic task completion success rate. |
+| Potential Known Risks | This model can generate synthetic media and may produce content that is offensive, unsafe, misleading, indecent, or unsuitable for a target deployment. Users should implement robust safety guardrails — including content filtering, abuse monitoring, and access controls — to reduce the risk of harmful outputs. Users are responsible for ensuring that their use of the model complies with all applicable laws and regulations, and for regularly reviewing and updating their guardrails as risks evolve. |
+| Licensing | [OpenMDW1.1](https://openmdw.ai/)  |

LICENSE

@@ -0,0 +1,49 @@
+OpenMDW License Agreement, version 1.1 (OpenMDW-1.1)
+
+By exercising rights granted to you under this agreement, you accept and agree
+to its terms.
+
+As used in this agreement, "Model Materials" means the materials provided to
+you under this agreement, consisting of: (1) one or more machine learning
+models (including architecture and parameters); and (2) all related artifacts
+(including associated data, documentation and software) that are provided to
+you hereunder.
+
+Subject to your compliance with this agreement, permission is hereby granted,
+free of charge, to deal in the Model Materials without restriction, including
+under all copyright, patent, database, and trade secret rights included or
+embodied therein.
+
+If you distribute any portion of the Model Materials, you shall retain in your
+distribution (1) a copy of this agreement, and (2) all copyright notices and
+other notices of origin included in the Model Materials that are applicable to
+your distribution.
+
+If you file, maintain, or voluntarily participate in a lawsuit against any
+person or entity asserting that the Model Materials directly or indirectly
+infringe any patent or copyright, then all rights and grants made to you
+hereunder are terminated, unless that lawsuit was in response to a
+corresponding lawsuit first brought against you.
+
+This agreement does not impose any restrictions or obligations with respect to
+any use, modification, or sharing of any outputs generated by using the Model
+Materials.
+
+THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE
+ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL
+TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
+
+YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY
+APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION
+ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL
+MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS
+REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE
+DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR
+ANYTHING INCORPORATED OR EMBODIED THEREIN.
+
+IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE
+USE THEREOF OR OTHER DEALINGS THEREIN.

PRIVACY.md

@@ -0,0 +1,6 @@
+## Privacy
+| Privacy Information |
+|---|
+| The model was trained on large-scale publicly available data that may contain images, audio-video, and text relating to people. NVIDIA collected and used this data in compliance with applicable data protection and privacy laws. This model was not designed to derive insights or otherwise learn from any personal data contained in the datasets. |
+| NVIDIA uses a combination of filters, data minimization techniques, and other guardrails to help prevent personal data from being recited by our models. We employ automated tools and data processing techniques during pre-training or training to identify and filter certain categories of personal data. For example, for text-bearing source and document components, our automated tools identified potential personal data such as person names, locations, and possible business or public-facing contact information such as email addresses and phone numbers.  We reviewed and removed any verified instances of personal data through a combination of automated filtering and human-in-the-loop validation. |
+| Please review NVIDIA's [Privacy Policy](https://www.nvidia.com/en-us/about-nvidia/privacy-policy/) for more information. |

README.md

@@ -0,0 +1,187 @@
+---
+license: other
+license_name: openmdw1.1-license
+license_link: >-
+  https://openmdw.ai/license/1-1/
+library_name: cosmos
+tags:
+  - nvidia
+  - cosmos
+  - cosmos3
+  - image-to-video
+  - video-generation
+  - nvfp4
+  - quantized
+  - modelopt
+---
+
+# Cosmos3-Super-Image2Video-4Step — NVFP4 (community quantization)
+
+This is a community **NVFP4, weight-only** quantization of NVIDIA's
+[`nvidia/Cosmos3-Super-Image2Video-4Step`](https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step) —
+a 64B-parameter image-to-video model, DMD2-distilled for 4-step, CFG-free
+generation — produced with [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer)
+(`nvidia-modelopt`). A sibling of
+[prometheusAIR/Cosmos3-Super-Image2Video-4Step-FP8](https://huggingface.co/prometheusAIR/Cosmos3-Super-Image2Video-4Step-FP8) —
+this format trades a bit more quantization error for a much smaller footprint:
+**~38GB resident vs FP8's ~66GB**, useful if you want headroom for longer clips,
+higher resolutions, or running alongside other GPU workloads on the same card.
+
+**Model Developer (original model): NVIDIA.** This is not an official NVIDIA
+release. Everything about the model itself — architecture, training data,
+benchmarks, full input/output specs, limitations, and ethical considerations —
+is unchanged from NVIDIA's original and is documented on
+[their model card](https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step);
+it isn't reproduced here. This card only covers what we changed (the
+quantization) and how to run *this* checkpoint.
+
+## License
+
+Released under the [OpenMDW-1.1](https://openmdw.ai/license/1-1/) license, same
+as the original model — see [`LICENSE`](LICENSE) in this repo.
+
+## Quantization Notes
+
+**Recipe:** NVFP4 (E2M1 4-bit weights, block_size 16, FP8/E4M3 block scales +
+an FP32 per-tensor scale) on the 64 transformer blocks' attention/MLP linears.
+Activations, embeddings, norms, the bundled Qwen3 reasoner head,
+`time_embedder`, and `proj_in`/`proj_out` are left in BF16. Weight-only, so no
+calibration forward pass was needed — same recipe and same spared layers as
+the FP8 sibling, just a smaller weight format. Per NVIDIA's own note, no
+quantized precision is officially tested/supported for this model — treat
+this as best-effort, and expect somewhat more quantization error than the
+FP8 release given the lower bit-width.
+
+**Loading gotchas — read before using with `diffusers`:** as of
+`diffusers==0.39.0.dev0`, plain `Cosmos3OmniPipeline` usage has **two** bugs
+against this checkpoint (identical to the FP8 release — same base checkpoint,
+same diffusers version, unrelated to which quantization format you use). Both
+are fixed automatically by the serving script below
+(`serve_cosmos3_i2v4step_diffusers.py`); if you're writing your own loading
+code, patch the scheduler yourself as shown.
+
+1. **Wrong step count / CFG on.** `Cosmos3OmniPipeline.__call__` does not read
+   this checkpoint's `scheduler/scheduler_config.json` →
+   `fixed_step_sampler_config.t_list` (the trained 4-step sde schedule). It
+   silently falls back to the pipeline's generic defaults
+   (`num_inference_steps=35`, `guidance_scale=6.0`), which is off-distribution
+   for a DMD2-distilled 4-step checkpoint. (`vLLM-Omni` parses this file
+   correctly; this gap is specific to the plain `diffusers` PyTorch path.)
+
+2. **Image conditioning gets destroyed into colorful static (critical).** This
+   checkpoint's scheduler ships `stochastic_sampling=True` (SDE sampling).
+   Cosmos3 anchors the conditioning frame by zeroing the model's predicted
+   velocity there — under a *deterministic* step that means "leave this frame
+   unchanged," but `FlowMatchEulerDiscreteScheduler`'s SDE branch computes
+   `x0 = sample - current_sigma * model_output` (= `sample`, since velocity is
+   0) then `prev_sample = (1 - next_sigma) * x0 + next_sigma * randn_tensor(...)`
+   — it re-noises by `next_sigma` **regardless of velocity**. Across this
+   checkpoint's 4 steps that compounds to ~99.6% fresh noise in the
+   conditioned frame: your input image comes out as colorful static while the
+   genuinely-denoised motion frames still look like plausible video (this
+   combination — garbage first frame, coherent-but-drifted rest — is exactly
+   how it presents). Disabling `stochastic_sampling` restores the correct
+   zero-velocity-is-a-no-op behavior. Confirmed by direct A/B render, same
+   seed/image/prompt, only this flag changed.
+
+```python
+def force_fixed_step_schedule(scheduler):
+    t_list = scheduler.config.fixed_step_sampler_config["t_list"]
+    orig = scheduler.set_timesteps
+    scheduler.set_timesteps = lambda num_inference_steps=None, device=None, **_: (
+        orig(sigmas=list(t_list), device=device)
+    )
+    if scheduler.config.stochastic_sampling:
+        scheduler.register_to_config(stochastic_sampling=False)
+
+pipe = ...  # Cosmos3OmniPipeline.from_pretrained(...)
+force_fixed_step_schedule(pipe.scheduler)
+
+result = pipe(prompt=..., image=..., guidance_scale=1.0, num_inference_steps=4, ...)
+# guidance_scale=1.0 disables CFG; num_inference_steps is a no-op once patched
+```
+
+Verified against this exact repo: the patch produces a 4-iteration denoising
+loop (not 35) with CFG off and a correctly-preserved conditioning frame
+(not colorful static), matching the checkpoint's trained regime. The
+save/restore round trip (`transformer/modelopt_state.pth`) reloads correctly
+with 896 quantized weight wrappers active.
+
+## Usage: Run Inference (single GPU, `diffusers`)
+
+This repo is a `diffusers`-loadable repackage (`transformer/modelopt_state.pth`),
+not NVIDIA's `vLLM-Omni` deployment export format. If you have a vLLM-Omni
+cluster, use NVIDIA's original BF16 checkpoint and card instead. For everyone
+else — anyone running this NVFP4 checkpoint on a single GPU — use the FastAPI
+server included in this repo, which loads the checkpoint, applies the 4-step
+scheduler fix above automatically, and exposes a plain HTTP endpoint.
+
+### 1. Install
+
+Requires a `diffusers` build with Cosmos3 support — not yet in a PyPI release
+as of this writing, so install from the exact commit this checkpoint was
+produced and validated against:
+
+```bash
+pip install "git+https://github.com/huggingface/diffusers.git@2c7efb95349296cf6bcce981ea036275a82a94df"
+pip install nvidia-modelopt accelerate torch fastapi uvicorn python-multipart
+```
+
+### 2. Download this repo (weights + scripts together)
+
+```bash
+hf download prometheusAIR/Cosmos3-Super-Image2Video-4Step-NVFP4 \
+    --local-dir Cosmos3-Super-Image2Video-4Step-NVFP4
+cd Cosmos3-Super-Image2Video-4Step-NVFP4
+```
+
+This pulls the whole ~39GB repo — NVFP4 weights, VAE, tokenizer, and the `.py`
+scripts side by side, the same download either way.
+
+### 3. Serve it
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python serve_cosmos3_i2v4step_diffusers.py --repo .
+```
+
+One GPU with roughly 45GB+ free memory is enough — noticeably less than the
+FP8 release (~70GB+), so this fits single 48GB-class cards (RTX 6000 Ada,
+A6000, L40S) in addition to the larger ones (RTX PRO 6000, H100/H200,
+A100-80GB). No multi-GPU sharding required. The server listens on
+`http://localhost:8000` once it's done loading.
+
+### 4. Use it
+
+```bash
+curl -s -X POST http://localhost:8000/animate \
+    -F image=@your_first_frame.png \
+    -F 'prompt=The robotic arm slowly lowers its gripper toward the objects and holds. Static camera.' \
+    -F num_frames=49 -F fps=24 \
+    --output clip.mp4
+```
+
+`num_frames` / `fps` / `height` / `width` are adjustable. `num_inference_steps`
+and `guidance_scale` are intentionally not exposed — this checkpoint's 4-step,
+CFG-free schedule is fixed and applied automatically by the server.
+
+### Other scripts in this repo
+
+- `quantize_cosmos3_i2v4step_streaming.py` — reproduces this NVFP4 quantization
+  (`--format nvfp4`) from NVIDIA's BF16 source checkpoint; also supports
+  `--format fp8` for the sibling release.
+- `repackage_for_hf_i2v4step.py` — rebuilds this diffusers-loadable repo format
+  from a quantized transformer.
+- `load_cosmos3_modelopt.py` — the underlying loader `serve_cosmos3_i2v4step_diffusers.py`
+  uses; import `load_pipe(...)` directly if you want a `pipe` object instead of
+  an HTTP server.
+- `validate_cosmos3_i2v4step_fp8.py` — a minimal standalone image→video smoke
+  test; despite the filename (kept for consistency with the FP8 repo) it
+  takes `--format nvfp4|fp8` and works against either checkpoint.
+
+## Responsible Use
+
+See NVIDIA's [Bias](BIAS.md), [Explainability](EXPLAINABILITY.md),
+[Safety & Security](SAFETY.md), and [Privacy](PRIVACY.md) subcards (included
+in this repo), and the Limitations / Ethical Considerations sections of
+[NVIDIA's original model card](https://huggingface.co/nvidia/Cosmos3-Super-Image2Video-4Step).
+Report security vulnerabilities [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

SAFETY.md

@@ -0,0 +1,11 @@
+## Safety & Security
+
+| Field | Response |
+| :---- | :---- |
+| Model Application(s) | World reasoning and generation for Physical AI. |
+| Describe the life critical impact: | This model is not a safety-certified component and must not be used as the sole basis for life-critical decisions or control without additional system-level validation, safety analysis, and safeguards. The model is not designed or tested by NVIDIA for use in any system or application where the use of or failure of such system or application developed with the model could result in injury, death, or catastrophic damage. NVIDIA is not liable to any party, in whole or in part, for any claims or damages arising from those uses. Any system or application developed with the model must include sufficient safety and redundancy features and comply with applicable legal and regulatory standards and requirements. |
+| Description of methods implemented in data acquisition or processing, if any, to address other types of potentially harmful data in the training, testing, and validation data:  | Training, evaluation, and validation datasets pass through multi-stage automated and manual filtering to reduce harmful, unsafe, restricted, or policy-violating content. Pipelines include source-licensing review, deduplication, metadata-based and classifier-based moderation, embedding-based anomaly detection, and human audits on selected datasets. For Physical AI data (robotics, autonomous driving, industrial scenes), filtering also targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark and red-team testing surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No data-filtering process can guarantee complete removal; developers are responsible for application-specific safeguards and validation before deployment. |
+| Description of any methods implemented in data acquisition or processing, if any, to address illegal or harmful content in the training data, including, but not limited to, child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII)  | In addition to the general unsafe-content filtering described above, training data acquisition and preprocessing apply CSAM- and NCII-specific safeguards: hash-matching systems against known CSAM databases, classifier-based moderation models trained specifically for explicit content and NCII detection, and provenance and licensing review for sources containing human imagery. Identified content is removed at ingest, with human review and targeted audits supplementing automated filtering for selected datasets. Despite these safeguards, no large-scale data-filtering system can guarantee complete detection. Ongoing monitoring and dataset review continue post-release. |
+| Use Case Restrictions | Use is governed by the [OpenMDW1.1](https://openmdw.ai/) |
+| Model and dataset restrictions | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to. |
+| Responsible Data Handling | This AI model was developed based on our policies to ensure responsible data handling and risk mitigation. The datasets used for training have been scanned for harmful content and illegal content, consistent with our policies including scanning for Child Sexual Abuse Material (CSAM). Ongoing review and monitoring mechanisms are in place based on our policies and to maintain data integrity. |

assets/example_output.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:625d490237c02ed103797bc42b6c2aff9b219779b0de29226df7878c671ec907
+size 8589231

assets/example_prompt.json

@@ -0,0 +1 @@
+{"prompt": "{\"temporal_caption\": \"A fallen waffle cone lies on rough sunlit asphalt in an extreme low close-up, with a rounded scoop of vanilla and chocolate ice cream pressed against the road, a small melted puddle already spreading beneath it, and dry autumn leaves scattered around in warm late-afternoon light. The viewpoint stays near ground level and begins a slow, smooth arc around the cone from left to right, keeping the melting scoop dominant while the background street and leaves remain softly blurred. As the sun warms the ice cream, the glossy edges soften first, thin rivulets of vanilla and chocolate slide down the curved scoop, and the existing puddle widens into the cracks and pebbled texture of the asphalt. The waffle cone remains mostly rigid but grows slightly damp at the rim touching the ice cream, while the scoop loses its rounded shape, slumps lower, and exposes more of the cone’s open mouth. The moving viewpoint continues its gentle orbit, revealing the chocolate side thinning into streaks and the vanilla side collapsing into pale liquid that creeps outward under gravity. By the end, most of the ice cream has flattened into a shallow glossy stain that drains into small road fissures and spreads out of the immediate area, leaving the cone lying in place with only thin cream-colored and brown traces clinging to the asphalt in the warm light.\", \"duration\": \"7s\", \"fps\": 24.0, \"resolution\": {\"H\": 480, \"W\": 832}, \"aspect_ratio\": \"16,9\"}", "negative_prompt": "The video captures a series of frames showing macroblocking artifacts, chromatic aberration, high-frequency noise, and rolling shutter distortion. It includes static with no motion, motion blur, over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, jerky movements, low frame rate, bit-depth compression artifacts, color banding, unnatural transitions, outdated special effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, hard cut, visual noise, and flickering. It features moiré patterns, edge halos, and temporal aliasing. Furthermore, the content defies common sense, generating illogical scenarios, nonsensical entities, absurd character behaviors, and conceptual paradoxes that violate basic human reasoning and everyday reality. The video looks like a surreal or glitchy hallucination. Overall, the video is of poor quality."}

assets/original_prompt.txt

@@ -0,0 +1 @@
+The ice cream melts and gradually disappears. The camera moves around.

chat_template.json

@@ -0,0 +1,3 @@
+{
+    "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' }}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- for message in messages %}\n    {%- if message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|vision_start|><|image_pad|><|vision_end|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|vision_start|><|video_pad|><|vision_end|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content_item in message.content %}\n                {%- if 'text' in content_item %}\n                    {{- content_item.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and message.content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|vision_start|><|image_pad|><|vision_end|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|vision_start|><|video_pad|><|vision_end|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n"
+}

checkpoint.json

@@ -0,0 +1,5 @@
+{
+  "checkpoint_type": "hf",
+  "source_weights": "ema",
+  "student_only": true
+}

config.json

@@ -0,0 +1,252 @@
+{
+  "allow_patterns_overrides": [
+    "*/*.safetensors"
+  ],
+  "architectures": [
+    "Cosmos3ForConditionalGeneration"
+  ],
+  "image_token_id": 151655,
+  "model": {
+    "_recursive_": false,
+    "_target": "omni_mot_model",
+    "config": {
+      "_type": "omni_mot_model_config",
+      "action_gen": false,
+      "activation_checkpointing": {
+        "_type": "activation_checkpointing_config",
+        "determinism_check": "default",
+        "mode": "full",
+        "preserve_rng_state": true,
+        "save_ops_regex": [
+          "fmha"
+        ]
+      },
+      "causal_training_strategy": "none",
+      "compile": {
+        "_type": "compile_config",
+        "compile_dynamic": true,
+        "compiled_region": "language",
+        "coordinate_descent_tuning": false,
+        "enabled": false,
+        "max_autotune_pointwise": false,
+        "use_cuda_graphs": false
+      },
+      "diffusion_expert_config": {
+        "_type": "diffusion_expert_config",
+        "base_fps": 16,
+        "enable_fps_modulation": true,
+        "load_weights_from_pretrained": true,
+        "max_vae_latent_side_after_patchify": 20,
+        "patch_spatial": 2,
+        "sound_base_temporal_compression_factor": null,
+        "timestep_range": 1.0,
+        "unified_3d_mrope_reset_spatial_ids": true,
+        "unified_3d_mrope_temporal_modality_margin": 15000,
+        "vision_temporal_position_mode": "latent_index"
+      },
+      "ema": {
+        "_type": "ema_config",
+        "enabled": false,
+        "iteration_shift": 0,
+        "rate": 0.1
+      },
+      "enable_input_bias": true,
+      "exclude_reasoner_weights_from_checkpoint": false,
+      "fixed_step_sampler_config": {
+        "_type": "fixed_step_sampler_config",
+        "sample_type": "sde",
+        "t_list": [
+          1.0,
+          0.9375,
+          0.8333333333333334,
+          0.625
+        ]
+      },
+      "input_caption_key": "ai_caption",
+      "input_image_key": "images",
+      "input_video_key": "video",
+      "joint_attn_implementation": "two_way",
+      "latent_downsample_factor": 16,
+      "lbl": {
+        "_type": "lbl_config",
+        "coeff_gen": null,
+        "coeff_und": null,
+        "method": "local"
+      },
+      "log_enc_time_every_n": 100,
+      "lora_alpha": 32,
+      "lora_enabled": false,
+      "lora_rank": 16,
+      "lora_target_modules": "q_proj_moe_gen,k_proj_moe_gen,v_proj_moe_gen,o_proj_moe_gen",
+      "max_action_dim": 32,
+      "max_num_tokens_after_packing": 45056,
+      "natten_parameter_list": null,
+      "net": null,
+      "num_embodiment_domains": 32,
+      "parallelism": {
+        "_type": "parallelism_config",
+        "attention_io_layout": "sequence_sharded",
+        "cfg_parallel_shard_degree": 1,
+        "context_parallel_shard_degree": 1,
+        "data_parallel_replicate_degree": 1,
+        "data_parallel_shard_degree": 128,
+        "enable_inference_mode": false,
+        "fsdp_master_dtype": "float32"
+      },
+      "precision": "bfloat16",
+      "rectified_flow_inference_config": {
+        "_type": "rectified_flow_inference_config",
+        "num_train_timesteps": 1000,
+        "scheduler_type": "unipc",
+        "shift": 1,
+        "use_dynamic_shifting": false
+      },
+      "rectified_flow_training_config": {
+        "_type": "rectified_flow_training_config",
+        "action_loss_weight": 10.0,
+        "image_loss_scale": null,
+        "independent_action_schedule": false,
+        "independent_sound_schedule": false,
+        "loss_scale": 10.0,
+        "normalize_loss_by_active": false,
+        "shift": {
+          "256": 3,
+          "480": 5,
+          "720": 10
+        },
+        "shift_action": null,
+        "shift_sound": null,
+        "sound_loss_scale": null,
+        "train_time_action_distribution": "logitnormal",
+        "train_time_image_distribution": "logitnormal",
+        "train_time_sound_distribution": "logitnormal",
+        "train_time_video_distribution": "uniform",
+        "train_time_weight": "uniform",
+        "use_discrete_rf": false,
+        "use_dynamic_shift": false
+      },
+      "resolution": "480",
+      "sound_dim": null,
+      "sound_gen": false,
+      "sound_latent_fps": 25,
+      "sound_tokenizer": null,
+      "state_ch": 48,
+      "state_t": 300,
+      "tokenizer": {
+        "_target": "wan2pt2_vae_interface",
+        "bucket_name": "bucket",
+        "causal": true,
+        "chunk_duration": 93,
+        "encode_bucket_multiple": 1,
+        "encode_chunk_frames": {
+          "256": 68,
+          "480": 24,
+          "720": 12
+        },
+        "encode_exact_durations": [
+          17,
+          61,
+          73
+        ],
+        "keep_decoder_cache": false,
+        "object_store_credential_path_pretrained": "",
+        "spatial_compression_factor": 16,
+        "temporal_compression_factor": 4,
+        "temporal_window": null,
+        "use_streaming_encode": false,
+        "vae_path": "pretrained/tokenizers/video/wan2pt2/Wan2.2_VAE.pth"
+      },
+      "video_temporal_causal": false,
+      "vision_gen": true,
+      "vlm_config": {
+        "_type": "vlm_config",
+        "layer_module": null,
+        "model_instance": {
+          "_target": "qwen3_vl_text_for_causal_lm",
+          "config": {
+            "_target": "create_vlm_config",
+            "base_config": {
+              "_target": "qwen3_vl_mot_config_from_json_file",
+              "json_file": "cosmos3://vfm/models/vlm/qwen3_vl/configs/Qwen3-VL-32B-Instruct.json"
+            },
+            "qk_norm_for_text": true
+          }
+        },
+        "model_name": "nvidia/Cosmos3-Super-Reasoner",
+        "pretrained_weights": {
+          "_type": "pretrained_weights_config",
+          "backbone_path": "",
+          "checkpoint_format": null,
+          "credentials_path": "",
+          "enable_gcs_patch_in_boto3": false,
+          "enabled": false
+        },
+        "qk_norm": false,
+        "safetensors_path": "",
+        "tie_word_embeddings": false,
+        "tokenizer": {
+          "_target": "create_qwen2_tokenizer_with_download",
+          "config_variant": "hf",
+          "pretrained_model_name": "Qwen/Qwen3-VL-32B-Instruct"
+        },
+        "use_system_prompt": false
+      }
+    }
+  },
+  "model_type": "cosmos3_omni",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "dtype": "bfloat16",
+    "eos_token_id": 151645,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 5120,
+    "initializer_range": 0.02,
+    "intermediate_size": 25600,
+    "max_position_embeddings": 262144,
+    "model_type": "qwen3_vl_text",
+    "num_attention_heads": 64,
+    "num_hidden_layers": 64,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        24,
+        20,
+        20
+      ],
+      "rope_type": "default"
+    },
+    "rope_theta": 5000000,
+    "use_cache": true,
+    "vocab_size": 151936
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.0.dev0",
+  "video_token_id": 151656,
+  "vision_config": {
+    "deepstack_visual_indexes": [
+      8,
+      16,
+      24
+    ],
+    "depth": 27,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "qwen3_vl",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 5120,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652
+}

generation_config.json

@@ -0,0 +1,14 @@
+{
+    "bos_token_id": 151643,
+    "pad_token_id": 151643,
+    "do_sample": true,
+    "eos_token_id": [
+        151645,
+        151643
+    ],
+    "top_p": 0.8,
+    "top_k": 20,
+    "temperature": 0.7,
+    "repetition_penalty": 1.0,
+    "transformers_version": "4.56.0"
+}

load_cosmos3_modelopt.py

@@ -0,0 +1,256 @@
+#!/usr/bin/env python
+"""
+Drop-in loader for the weight-only NVFP4 / FP8 (NVIDIA ModelOpt) Cosmos3-Super checkpoint
+saved in the ROUND-TRIPPABLE format (modelopt_state.pth present -- see repackage_for_hf.py).
+
+Current diffusers + accelerate + modelopt treat this path as experimental; each shim below
+works around a specific, source-verified version-skew gap. None modify the checkpoint:
+
+  1. enable_huggingface_checkpointing()
+     Registers ModelOpt's HF handlers so `from_pretrained` restores the quantized module
+     structure from modelopt_state.pth before weights load.
+
+  2. set_module_tensor_to_device patch (parameter materialization)
+     The modelopt_state restore runs inside diffusers' meta-device init, so each quantized
+     weight is a QTensorWrapper whose storage is a META tensor carrying dequant metadata.
+     `param.data = real` can't cross meta<->real, and accelerate's rebuild both rejects
+     requires_grad and discards metadata. We REPLACE the parameter with a fresh wrapper
+     around the loaded bytes + the existing metadata. Two extra duties here:
+       - payload dtype restore: diffusers casts floating params to torch_dtype during load
+         when no hf_quantizer is present (model_loading_utils.py -- their own TODO flags
+         float8). FP8 payloads are floating and arrive cast to bf16; we cast back to the
+         wrapper's payload dtype (exact: every e4m3fn value round-trips through bf16).
+         NVFP4's uint8 payload is never cast, so this is a no-op there.
+       - direct-to-GPU materialization: staging payloads on CPU would put the whole packed
+         model in system RAM (FP8: ~64 GB on a 32 GB box -> OOM-killed; NVFP4 only survived
+         because uncast tensors stay mmap-backed). Wrappers go straight to `materialize_device`.
+
+  3. Post-restore quantizer re-disable
+     modelopt_state replays the QUANT CONFIG, not imperative `.disable()` calls made after
+     quantize. The NVFP4 build used NVFP4_DEFAULT_CFG (activation quantization ON in-config)
+     with activations disabled imperatively -- so the restored model comes back with ~1806
+     dynamic fake-quant activation quantizers active: ~10x slower, fatter, and quantizing
+     activations the validated regime never quantized. We re-apply weight-only + spare
+     disabling after load. (FP8's config had the disables baked in; this is then a no-op.)
+
+  4. Full bf16 normalization (the validated serve regime)
+     Cosmos3OmniTransformer pins time_embedder to fp32 via _keep_in_fp32_modules
+     (transformer_cosmos3.py:297); the quantized model runs all-bf16. Cast floating buffers
+     and non-wrapper floating params to bf16, retarget wrapper dequant dtype to bf16, and
+     register bf16 input-cast pre-hooks on the time embedders.
+
+  5. NVIDIAModelOptQuantizer.create_quantized_param patch
+     Only relevant if a checkpoint carries an embedded quantization_config (hf_quantizer
+     path): requires_grad only for float/complex tensors. Kept for robustness.
+
+Usage (library):
+    from load_cosmos3_modelopt import load_pipe
+    pipe = load_pipe("YOUR_HF_USERNAME/Cosmos3-Super-nvfp4")   # or a local dir
+    # NOTE: a bare pipe(prompt) call renders the pipeline DEFAULT: a 189-frame 720x1280
+    # video (~8 s at 24 fps) -- not a still. For a single-image smoke test, be explicit:
+    r = pipe("a red cube on a table", height=1024, width=1024, num_frames=1,
+             num_inference_steps=50, guidance_scale=4.0)
+    r.video[0].save("out.png")          # .video is the list of PIL frames; [0] is the image
+
+Usage (interactive -- lands in a REPL with `pipe` loaded):
+    CUDA_VISIBLE_DEVICES=0 python -i load_cosmos3_modelopt.py ./cosmos3-super-nvfp4-hf
+"""
+import os
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch
+
+SPARE_SUBSTRINGS = [
+    "time_embedder", "proj_in", "proj_out", "lm_head", "embed", "norm", "audio_proj",
+]
+
+
+def _qtensor_wrapper_cls():
+    try:
+        from modelopt.torch.quantization.qtensor.base_qtensor import QTensorWrapper
+        return QTensorWrapper
+    except Exception:
+        return None
+
+
+def _patch_modelopt_quantizer() -> None:
+    """diffusers pre-quantized hf_quantizer branch: only float/complex tensors may require grad."""
+    import diffusers.quantizers.modelopt.modelopt_quantizer as moq
+
+    if getattr(moq.NVIDIAModelOptQuantizer, "_cqp_patched", False):
+        return
+    _orig = moq.NVIDIAModelOptQuantizer.create_quantized_param
+
+    def _cqp(self, model, param_value, param_name, target_device, *args, **kwargs):
+        if self.pre_quantized:
+            module, tname = moq.get_module_from_name(model, param_name)
+            needs_grad = param_value.is_floating_point() or param_value.is_complex()
+            module._parameters[tname] = torch.nn.Parameter(
+                param_value.to(device=target_device), requires_grad=needs_grad
+            )
+            return
+        return _orig(self, model, param_value, param_name, target_device, *args, **kwargs)
+
+    moq.NVIDIAModelOptQuantizer.create_quantized_param = _cqp
+    moq.NVIDIAModelOptQuantizer._cqp_patched = True
+
+
+def _patch_qtensor_loading() -> None:
+    """Materialize restored (meta) QTensorWrapper params: replace the parameter object with
+    a fresh wrapper around the loaded bytes, restoring the payload dtype (undoes diffusers'
+    float cast on FP8) and landing directly on the target device (keeps payloads out of RAM)."""
+    import diffusers.models.model_loading_utils as mlu
+
+    if getattr(mlu, "_qtw_inplace_patched", False):
+        return
+    QTensorWrapper = _qtensor_wrapper_cls()
+    if QTensorWrapper is None:
+        return  # different modelopt layout; nothing to patch
+
+    _orig = mlu.set_module_tensor_to_device
+    stats = {"materialized": 0, "target_device": None}
+
+    def _patched(model, tensor_name, device, value=None, *args, **kwargs):
+        if value is not None:
+            module, leaf = model, tensor_name
+            if "." in tensor_name:
+                mod_path, leaf = tensor_name.rsplit(".", 1)
+                try:
+                    module = model.get_submodule(mod_path)
+                except AttributeError:
+                    module = None
+            if module is not None:
+                cur = getattr(module, "_parameters", {}).get(leaf)
+                if isinstance(cur, QTensorWrapper):
+                    tgt = stats.get("target_device") or device
+                    module._parameters[leaf] = QTensorWrapper(
+                        value.to(device=tgt, dtype=cur.data.dtype),  # exact cast-back for fp8
+                        metadata=dict(cur.metadata),
+                    )
+                    stats["materialized"] += 1
+                    return None
+        return _orig(model, tensor_name, device, value=value, *args, **kwargs)
+
+    mlu.set_module_tensor_to_device = _patched
+    mlu._qtw_inplace_patched = True
+    mlu._qtw_stats = stats
+
+
+def _enforce_weight_only(transformer) -> None:
+    """Re-apply the validated weight-only + spare regime: the state replay re-enables any
+    quantizers that were disabled imperatively after quantize (NVFP4 default cfg has
+    activation quantization ON in-config)."""
+    n_act = n_spare = 0
+    for name, m in transformer.named_modules():
+        if not (name.endswith("_quantizer") and hasattr(m, "disable")):
+            continue
+        if name.endswith("weight_quantizer"):
+            if any(s in name for s in SPARE_SUBSTRINGS):
+                if getattr(m, "is_enabled", False):
+                    n_spare += 1
+                m.disable()
+        else:
+            if getattr(m, "is_enabled", False):
+                n_act += 1
+            m.disable()
+    print(f"[load] re-disabled quantizers the state replay re-enabled: "
+          f"{n_act} activation, {n_spare} spare-weight")
+
+
+def _apply_dtype_nudges(transformer) -> None:
+    """Reproduce the validated all-bf16 serve regime (all no-ops where already aligned)."""
+    QTensorWrapper = _qtensor_wrapper_cls() or ()
+
+    n_buf = n_par = n_meta = 0
+    for m in transformer.modules():
+        for bn, buf in list(m._buffers.items()):
+            if buf is not None and buf.is_floating_point() and buf.dtype != torch.bfloat16:
+                m._buffers[bn] = buf.to(torch.bfloat16)
+                n_buf += 1
+    for _, p in transformer.named_parameters():
+        if isinstance(p, QTensorWrapper):
+            d = p.metadata.get("dtype")
+            if isinstance(d, torch.dtype) and d.is_floating_point and d != torch.bfloat16:
+                p.metadata["dtype"] = torch.bfloat16   # dequant target only; payload untouched
+                n_meta += 1
+            continue  # packed payloads: never cast
+        if p.is_floating_point() and p.dtype != torch.bfloat16:
+            p.data = p.data.to(torch.bfloat16)
+            n_par += 1
+    print(f"[load] normalized to bf16: {n_par} params, {n_buf} buffers; "
+          f"retargeted {n_meta} dequant dtypes")
+
+    def _cast_bf16(_m, args):
+        return tuple(
+            a.to(torch.bfloat16)
+            if torch.is_tensor(a) and a.is_floating_point() and a.dtype != torch.bfloat16
+            else a
+            for a in args
+        )
+
+    for name, m in transformer.named_modules():
+        if "time_embedder" in name and hasattr(m, "linear_1"):
+            m.register_forward_pre_hook(_cast_bf16)
+
+
+def load_pipe(
+    model_id_or_path: str,
+    *,
+    torch_dtype=torch.bfloat16,
+    enable_safety_checker: bool = False,
+    device: str = "cuda",
+    materialize_device: str | None = "cuda",   # packed weights stream straight here (RAM stays low)
+    **kwargs,
+):
+    """Load a ModelOpt-quantized Cosmos3-Super pipeline with all load-time fixes applied."""
+    from diffusers import Cosmos3OmniPipeline
+    from modelopt.torch.opt import enable_huggingface_checkpointing
+
+    enable_huggingface_checkpointing()   # must run before from_pretrained
+    _patch_modelopt_quantizer()
+    _patch_qtensor_loading()
+
+    import diffusers.models.model_loading_utils as mlu
+    if hasattr(mlu, "_qtw_stats"):
+        mlu._qtw_stats["materialized"] = 0
+        mlu._qtw_stats["target_device"] = materialize_device
+
+    pipe = Cosmos3OmniPipeline.from_pretrained(
+        model_id_or_path,
+        torch_dtype=torch_dtype,
+        enable_safety_checker=enable_safety_checker,
+        **kwargs,
+    )
+
+    n_mat = getattr(mlu, "_qtw_stats", {}).get("materialized", 0)
+    print(f"[load] materialized {n_mat} packed quantized weight tensors")
+
+    transformer = getattr(pipe, "transformer", None)
+    if transformer is not None:
+        _enforce_weight_only(transformer)
+        _apply_dtype_nudges(transformer)
+
+    pipe = pipe.to(device)
+
+    QTensorWrapper = _qtensor_wrapper_cls()
+    if QTensorWrapper is not None and transformer is not None:
+        n_live = sum(1 for p in transformer.parameters() if isinstance(p, QTensorWrapper))
+        print(f"[load] {n_live} quantized weight wrappers active after move to {device}")
+        if n_mat and not n_live:
+            print("[load] WARNING: wrappers were lost during .to() -- do not render; report this")
+
+    return pipe
+
+
+if __name__ == "__main__":
+    import sys
+
+    path = sys.argv[1] if len(sys.argv) > 1 else "./cosmos3-super-nvfp4-hf"
+    print(f"[load] loading {path} ...")
+    pipe = load_pipe(path)
+    print("[load] OK -- `pipe` is ready.")
+    print("       NOTE: bare pipe(prompt) renders a 189-frame 720x1280 video (pipeline default).")
+    print("       single-still smoke test:")
+    print("       r = pipe('a red cube on a table', height=1024, width=1024, num_frames=1,")
+    print("                num_inference_steps=50, guidance_scale=4.0); r.video[0].save('out.png')")

model_index.json

@@ -0,0 +1,28 @@
+{
+  "_class_name": "Cosmos3OmniPipeline",
+  "_diffusers_version": "0.39.0",
+  "safety_checker": [
+    null,
+    null
+  ],
+  "scheduler": [
+    "diffusers",
+    "FlowMatchEulerDiscreteScheduler"
+  ],
+  "sound_tokenizer": [
+    null,
+    null
+  ],
+  "text_tokenizer": [
+    "transformers",
+    "Qwen2TokenizerFast"
+  ],
+  "transformer": [
+    "diffusers",
+    "Cosmos3OmniTransformer"
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKLWan"
+  ]
+}

modular_model_index.json

@@ -0,0 +1,64 @@
+{
+  "_blocks_class_name": "Cosmos3DistilledBlocks",
+  "_class_name": "Cosmos3DistilledModularPipeline",
+  "_diffusers_version": "0.39.0",
+  "is_distilled": true,
+  "distilled_sigmas": [
+    1.0,
+    0.9375,
+    0.8333333333333334,
+    0.625
+  ],
+  "scheduler": [
+    "diffusers",
+    "FlowMatchEulerDiscreteScheduler",
+    {
+      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Image2Video-4Step",
+      "subfolder": "scheduler",
+      "type_hint": [
+        "diffusers",
+        "FlowMatchEulerDiscreteScheduler"
+      ],
+      "variant": null
+    }
+  ],
+  "text_tokenizer": [
+    "transformers",
+    "Qwen2TokenizerFast",
+    {
+      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Image2Video-4Step",
+      "subfolder": "text_tokenizer",
+      "type_hint": [
+        "transformers",
+        "Qwen2TokenizerFast"
+      ],
+      "variant": null
+    }
+  ],
+  "transformer": [
+    "diffusers",
+    "Cosmos3OmniTransformer",
+    {
+      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Image2Video-4Step",
+      "subfolder": "transformer",
+      "type_hint": [
+        "diffusers",
+        "Cosmos3OmniTransformer"
+      ],
+      "variant": null
+    }
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKLWan",
+    {
+      "pretrained_model_name_or_path": "nvidia/Cosmos3-Super-Image2Video-4Step",
+      "subfolder": "vae",
+      "type_hint": [
+        "diffusers",
+        "AutoencoderKLWan"
+      ],
+      "variant": null
+    }
+  ]
+}

preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 16777216,
+        "shortest_edge": 65536
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "image_processor_type": "Qwen2VLImageProcessorFast"
+}

quantize_cosmos3_i2v4step_streaming.py

@@ -0,0 +1,383 @@
+#!/usr/bin/env python
+"""
+Stream-quantize NVIDIA Cosmos3-Super-Image2Video-4Step's transformer to weight-only
+FP8 or NVFP4, WITHOUT ever materializing the full ~128 GB BF16 model.
+
+This is the same-architecture sibling of quantize_cosmos3_super_streaming.py (the
+recipe validated against nvidia/Cosmos3-Super): identical Cosmos3OmniTransformer
+class, identical dims (64 layers, hidden_size 5120), so the empty-on-meta ->
+mtq.quantize -> mtq.compress -> stream-load recipe and the SPARE_SUBSTRINGS list
+carry over unchanged. Three things do NOT carry over -- this checkpoint is a DMD2
+4-step distillation of Cosmos3-Super-Image2Video, and differs as follows:
+
+  1. SCHEDULER IS FIXED, NOT SWAPPABLE. The base-model script (and
+     serve_cosmos3_diffusers.py) replace the shipped scheduler with
+     UniPCMultistepScheduler(flow_shift=...). This checkpoint ships
+     FlowMatchEulerDiscreteScheduler with a baked-in fixed_step_sampler_config
+     (sde sampling, t_list=[1.0, 0.9375, 0.8333, 0.625] -- literally 4 steps).
+     Do NOT override it -- use pipe.scheduler as loaded.
+  2. num_inference_steps / guidance_scale are NOT meaningfully configurable per
+     NVIDIA's model card (CFG is distilled out; the checkpoint was trained for an
+     exact 4-step sde schedule). CONFIRMED BY RUNNING IT: Cosmos3OmniPipeline
+     (diffusers 0.39.0.dev0) does not know this checkpoint's scheduler carries a
+     fixed_step_sampler_config -- an unpatched call silently ran the pipeline's
+     35-step/guidance=6.0 default instead. render_from_memory() below patches the
+     scheduler (_force_fixed_step_schedule) to force the correct 4-step t_list and
+     passes guidance_scale=1.0 explicitly. See that function's docstring.
+  3. This checkpoint is Image2Video-specific (unlike the T2I/T2V/I2V-omni base
+     model), so the smoke render requires a conditioning IMAGE -- there is no
+     text-only still-image path to fall back on. sound_tokenizer is also null in
+     this repo (audio branch inactive), so the "audio_proj" spare substring is
+     believed inert here, but is kept for parity/safety.
+
+USAGE
+-----
+    python quantize_cosmos3_i2v4step_streaming.py --format fp8
+    python quantize_cosmos3_i2v4step_streaming.py --format fp8 --smoke --image out.png
+
+Outputs go to ./cosmos3-i2v4step-<fmt>/ (override with --export-dir).
+"""
+
+import argparse
+import os
+
+# Reduce allocator fragmentation on the big card (cheap, always-on).
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch
+from accelerate import init_empty_weights, load_checkpoint_in_model
+from accelerate.utils import get_max_memory, infer_auto_device_map
+from accelerate.utils.dataclasses import CustomDtype
+from huggingface_hub import snapshot_download
+
+import modelopt.torch.quantization as mtq
+from modelopt.torch.export import export_hf_checkpoint
+
+# Cosmos3 classes require diffusers built from git main (already installed in the venv).
+from diffusers import Cosmos3OmniTransformer
+
+SRC_REPO = "nvidia/Cosmos3-Super-Image2Video-4Step"
+
+# ---------------------------------------------------------------------------
+# Layers to KEEP IN BF16 (never quantize). Matched as substrings of module names.
+# Unchanged from the base-model recipe -- same transformer class + dims, so the
+# same layers are quality-sensitive: embeddings, norms, the bundled Qwen3 reasoner
+# head, time/modality adapters, and the in/out projections.
+# ---------------------------------------------------------------------------
+SPARE_SUBSTRINGS = [
+    "time_embedder",
+    "proj_in",
+    "proj_out",
+    "lm_head",
+    "embed",        # token / position embeddings
+    "norm",         # layernorms / rmsnorms
+    "audio_proj",   # audio modality adapter (inert here: sound_tokenizer is null)
+]
+
+
+def _is_spare(module_name: str) -> bool:
+    return any(s in module_name for s in SPARE_SUBSTRINGS)
+
+
+def build_quant_cfg(fmt: str) -> dict:
+    """Return a WEIGHT-ONLY quant config for the chosen format. Same as the
+    base-model recipe: FP8 per-tensor E4M3, or NVFP4 4-bit block-scale."""
+    if fmt == "fp8":
+        return {
+            "quant_cfg": {
+                "*weight_quantizer": {"num_bits": (4, 3), "axis": None, "enable": True},
+                "*input_quantizer": {"enable": False},
+                "*output_quantizer": {"enable": False},
+                "*softmax_quantizer": {"enable": False},
+            },
+            "algorithm": "max",
+        }
+    elif fmt == "nvfp4":
+        import copy
+
+        base = getattr(mtq, "W4A16_NVFP4_CFG", None) or mtq.NVFP4_DEFAULT_CFG
+        return copy.deepcopy(base)
+    else:
+        raise ValueError(f"Unknown format: {fmt!r}")
+
+
+def enforce_weight_only_and_spare(model) -> tuple[int, int]:
+    """Disable all activation quantizers, and weight quantizers on SPARE layers.
+    Config-form agnostic (works for both the FP8 dict and modelopt's NVFP4 preset)."""
+    n_spare = 0
+    n_act = 0
+    for name, module in model.named_modules():
+        if not (name.endswith("_quantizer") and hasattr(module, "disable")):
+            continue
+        if name.endswith("weight_quantizer"):
+            parent = name.rsplit(".", 1)[0]
+            if _is_spare(parent):
+                module.disable()
+                n_spare += 1
+        else:
+            module.disable()
+            n_act += 1
+    return n_spare, n_act
+
+
+def compressed_device_map(model, gpu_mem_fraction: float = 0.85) -> dict:
+    """Build a device_map sized for the COMPRESSED weights (same as base-model recipe)."""
+    max_memory = {k: v * gpu_mem_fraction for k, v in get_max_memory().items()}
+
+    no_split = set()
+    for name, module in model.named_modules():
+        if name.endswith((".layers.0", ".blocks.0", ".transformer_blocks.0")):
+            no_split.add(module.__class__.__name__)
+
+    special_dtypes = {}
+    for name, module in model.named_modules():
+        if (
+            hasattr(module, "weight")
+            and hasattr(module, "weight_quantizer")
+            and getattr(module.weight_quantizer, "is_enabled", True)
+            and not getattr(module.weight_quantizer, "fake_quant", True)
+        ):
+            nb = module.weight_quantizer.num_bits
+            if isinstance(nb, tuple):
+                nb = nb[0] + nb[1] + 1
+            special_dtypes[name + ".weight"] = CustomDtype.FP8 if nb == 8 else CustomDtype.INT4
+
+    return infer_auto_device_map(
+        model,
+        max_memory=max_memory,
+        no_split_module_classes=list(no_split),
+        special_dtypes=special_dtypes,
+    )
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--format", choices=["fp8", "nvfp4"], required=True,
+                    help="Weight-only quantization format to produce.")
+    ap.add_argument("--export-dir", default=None,
+                    help="Output dir (default: ./cosmos3-i2v4step-<format>).")
+    ap.add_argument("--gpu-mem-fraction", type=float, default=0.85,
+                    help="Fraction of each GPU's memory accelerate may use for placement.")
+    ap.add_argument("--smoke", action="store_true",
+                    help="Validate by rendering a short clip from the in-memory model "
+                         "(runs before export; output: cosmos3_i2v4step_<fmt>_validate.mp4).")
+    ap.add_argument("--image", default="out.png",
+                    help="Conditioning image for the --smoke render (this checkpoint is "
+                         "Image2Video-only -- there is no text-only still path).")
+    args = ap.parse_args()
+
+    export_dir = args.export_dir or f"./cosmos3-i2v4step-{args.format}"
+    os.makedirs(export_dir, exist_ok=True)
+
+    print(f"[1/6] Resolving local checkpoint for {SRC_REPO} (transformer only)...")
+    local_root = snapshot_download(SRC_REPO, allow_patterns=["transformer/*"])
+    transformer_dir = os.path.join(local_root, "transformer")
+    print(f"      transformer dir: {transformer_dir}")
+
+    print("[2/6] Building EMPTY transformer on meta device (params on meta, buffers real)...")
+    config = Cosmos3OmniTransformer.load_config(transformer_dir)
+    with init_empty_weights(include_buffers=False):
+        model = Cosmos3OmniTransformer.from_config(config)
+
+    print(f"[3/6] Inserting quantizers ({args.format}) on the meta model...")
+    quant_cfg = build_quant_cfg(args.format)
+    mtq.quantize(model, quant_cfg)  # no forward_loop: weight-only needs no calibration
+    n_spare, n_act = enforce_weight_only_and_spare(model)
+    print(f"      weight-only: disabled {n_act} activation quantizers; "
+          f"kept {n_spare} projection layers in BF16 (plus embeddings/norms/head)")
+
+    print("[4/6] Setting up compressed parameter shapes (mtq.compress)...")
+    try:
+        mtq.compress(model, config=mtq.CompressConfig(quant_gemm=False))
+    except (AttributeError, TypeError):
+        mtq.compress(model)
+
+    print("[5/6] Streaming BF16 shards into compressed form (this is the long step)...")
+    device_map = compressed_device_map(model, args.gpu_mem_fraction)
+    load_checkpoint_in_model(
+        model,
+        checkpoint=transformer_dir,
+        device_map=device_map,
+        dtype=torch.bfloat16,
+    )
+
+    n_fixed = 0
+    for _, module in model.named_modules():
+        for bname, buf in list(module._buffers.items()):
+            if buf is not None and getattr(buf, "is_meta", False):
+                module._buffers[bname] = torch.zeros(buf.shape, dtype=buf.dtype, device="cuda")
+                n_fixed += 1
+        for pname, par in list(module._parameters.items()):
+            if par is not None and getattr(par, "is_meta", False):
+                module._parameters[pname] = torch.nn.Parameter(
+                    torch.zeros(par.shape, dtype=par.dtype, device="cuda"), requires_grad=False
+                )
+                n_fixed += 1
+    if n_fixed:
+        print(f"      materialized {n_fixed} residual meta tensors (disabled-quantizer scratch)")
+
+    n_bytes = sum(p.numel() * p.element_size() for p in model.parameters() if p.device.type != "meta")
+    n_bytes += sum(b.numel() * b.element_size() for b in model.buffers() if b.device.type != "meta")
+    print(f"      live footprint: {n_bytes / 1e9:.1f} GB")
+
+    if args.smoke:
+        render_from_memory(model, args.format, args.image)
+
+    print(f"[6/6] Exporting unified HF checkpoint to {export_dir} ...")
+    with torch.inference_mode():
+        export_hf_checkpoint(model, export_dir=export_dir)
+    print(f"DONE. Quantized {args.format.upper()} checkpoint written to {export_dir}")
+
+
+def _force_fixed_step_schedule(scheduler) -> bool:
+    """CONFIRMED (not speculative): Cosmos3OmniPipeline.__call__ always calls
+    scheduler.set_timesteps(num_inference_steps, device=device) with no sigmas
+    passthrough (pipeline_cosmos3_omni.py:1529 in diffusers 0.39.0.dev0), so this
+    checkpoint's scheduler_config.json fixed_step_sampler_config (t_list) is silently
+    dropped -- an unpatched call falls back to the pipeline's num_inference_steps=35
+    default (verified empirically: a real run produced a 35-step progress bar despite
+    this being a 4-step DMD2-distilled checkpoint). Running any schedule other than the
+    exact trained t_list is off-distribution for what the model was distilled to
+    denoise, so patch THIS scheduler instance's set_timesteps to always use it,
+    regardless of whatever num_inference_steps the pipeline passes in.
+
+    ALSO disables stochastic_sampling (forces the deterministic ODE branch) -- CONFIRMED
+    by direct A/B render, not speculative. This checkpoint ships stochastic_sampling=True
+    (SDE sampling). Cosmos3's image-conditioning anchors frame 0 by zeroing the model's
+    predicted velocity there (_mask_velocity_predictions), which only means "leave this
+    position unchanged" under the DETERMINISTIC step (prev_sample = sample + dt*0 =
+    sample). The SDE branch instead computes
+        x0 = sample - current_sigma * model_output   # = sample when velocity is 0
+        prev_sample = (1 - next_sigma) * x0 + next_sigma * randn_tensor(...)
+    which re-noises the sample by next_sigma regardless of velocity -- there is no
+    zero-velocity no-op in the SDE formula. Across this checkpoint's 4 steps that
+    compounds to ~99.6% fresh noise by the final step: the conditioning image is
+    destroyed into colorful static while the (non-anchored, genuinely denoised) motion
+    frames still look like a plausible video. Disabling stochastic_sampling makes zero
+    velocity a true no-op again, which an A/B render (same seed/image/prompt) confirmed
+    fixes it outright. This is presumably NVIDIA's own reference runtime doing a masked
+    resample this diffusers pipeline port doesn't implement; revisit if that lands
+    upstream."""
+    cfg = getattr(scheduler.config, "fixed_step_sampler_config", None)
+    t_list = cfg.get("t_list") if isinstance(cfg, dict) else None
+    if not t_list:
+        print("[warn] no fixed_step_sampler_config.t_list on this scheduler; leaving set_timesteps unpatched")
+        return False
+    _orig = scheduler.set_timesteps
+
+    def _patched(num_inference_steps=None, device=None, sigmas=None, mu=None, timesteps=None):
+        return _orig(sigmas=list(t_list), device=device)
+
+    scheduler.set_timesteps = _patched
+    if getattr(scheduler.config, "stochastic_sampling", False):
+        scheduler.register_to_config(stochastic_sampling=False)
+        print("[patch] disabled stochastic_sampling (SDE re-noising corrupts the "
+              "image-conditioned frame -- see docstring)")
+    print(f"[patch] forced fixed {len(t_list)}-step sde schedule: {t_list}")
+    return True
+
+
+def render_from_memory(model, fmt: str, image_path: str):
+    """Validate the quantization by rendering an image->video clip from the IN-MEMORY
+    model. Does NOT swap the scheduler object itself (keeps FlowMatchEulerDiscreteScheduler
+    with its stochastic_sampling config), but DOES patch its set_timesteps via
+    _force_fixed_step_schedule -- see that function for why this is required rather than
+    optional. guidance_scale=1.0 is passed explicitly: CFG is distilled out of this
+    checkpoint (do_classifier_free_guidance is guidance_scale != 1.0 in the pipeline
+    source), and the 6.0 pipeline default would silently re-enable it."""
+    import gc
+
+    print(f"\n[validate] Rendering an image->video clip from the in-memory {fmt.upper()} model "
+          f"(conditioning image: {image_path})...")
+    try:
+        from PIL import Image
+
+        from diffusers import Cosmos3OmniPipeline
+
+        if not os.path.isfile(image_path):
+            raise FileNotFoundError(
+                f"conditioning image {image_path!r} not found -- pass --image with a real "
+                "file; this checkpoint has no text-only still path to fall back on."
+            )
+        cond_image = Image.open(image_path).convert("RGB")
+
+        model.to("cuda")
+
+        # Dtype consistency at render time (same nudge as the base-model script): our
+        # model mixes FP8/NVFP4 (compressed) weights with BF16 spare weights, while
+        # diffusers computes the timestep sinusoidal fresh in FP32 each step.
+        for _module in model.modules():
+            for _bn, _buf in list(_module._buffers.items()):
+                if _buf is not None and _buf.dtype == torch.float32:
+                    _module._buffers[_bn] = _buf.to(torch.bfloat16)
+
+        def _cast_inputs_bf16(_m, args):
+            return tuple(
+                a.to(torch.bfloat16)
+                if torch.is_tensor(a) and a.is_floating_point() and a.dtype != torch.bfloat16
+                else a
+                for a in args
+            )
+
+        n_hooks = 0
+        for _name, _module in model.named_modules():
+            if "time_embedder" in _name and hasattr(_module, "linear_1"):
+                _module.register_forward_pre_hook(_cast_inputs_bf16)
+                n_hooks += 1
+        print(f"[validate] dtype-safety: cast fp32 buffers to bf16, hooked {n_hooks} time-embedder(s)")
+
+        # Pass OUR quantized transformer in so the pipeline does NOT reload it from the hub.
+        # Deliberately do NOT touch pipe.scheduler -- the shipped FlowMatchEulerDiscreteScheduler
+        # carries the fixed 4-step sde schedule for this checkpoint.
+        pipe = Cosmos3OmniPipeline.from_pretrained(
+            SRC_REPO,
+            transformer=model,
+            torch_dtype=torch.bfloat16,
+            enable_safety_checker=False,
+        )
+        for name, comp in pipe.components.items():
+            if name != "transformer" and isinstance(comp, torch.nn.Module):
+                comp.to("cuda")
+
+        _force_fixed_step_schedule(pipe.scheduler)
+
+        prompt = (
+            "The camera holds static as the scene continues naturally, with subtle "
+            "motion and realistic physics."
+        )
+        with torch.inference_mode():
+            # num_inference_steps is a no-op once _force_fixed_step_schedule has patched
+            # set_timesteps (it always substitutes the checkpoint's own t_list), but is
+            # passed for clarity. guidance_scale=1.0 disables CFG -- see docstring above.
+            # 13 frames (4n+1) keeps the smoke test quick; bump for a real quality check.
+            result = pipe(
+                prompt=prompt,
+                negative_prompt="",
+                image=cond_image,
+                num_frames=13,
+                height=256,
+                width=256,
+                num_inference_steps=4,
+                guidance_scale=1.0,
+                generator=torch.Generator(device="cuda").manual_seed(1234),
+            )
+        out_path = f"cosmos3_i2v4step_{fmt}_validate.mp4"
+        from diffusers.utils import export_to_video
+        export_to_video(result.video, out_path, fps=24)
+        print(f"[validate] Wrote {out_path}. Eyeball it for coherence with the conditioning image.")
+
+        del pipe, result
+        gc.collect()
+        torch.cuda.empty_cache()
+    except Exception as e:
+        import traceback
+        print(f"[validate] Render failed ({type(e).__name__}: {e}).")
+        print("[validate] This does NOT affect the quantized weights; export still proceeds below.")
+        print("[validate] If this is a TypeError about an unexpected keyword argument, that's real "
+              "signal about this checkpoint's actual pipe() call shape -- adjust the call above "
+              "(e.g. drop/rename kwargs) rather than assuming the defaults here were correct.")
+        traceback.print_exc()
+
+
+if __name__ == "__main__":
+    main()

repackage_for_hf_i2v4step.py

@@ -0,0 +1,97 @@
+#!/usr/bin/env python
+"""
+Re-save the in-memory quantized Cosmos3-Super-Image2Video-4Step transformer in the
+ROUND-TRIPPABLE ModelOpt HF format and assemble a complete drop-in diffusers repo
+around it. Sibling of repackage_for_hf.py -- identical logic, just imports the
+build helpers from serve_cosmos3_i2v4step_diffusers instead (which targets
+nvidia/Cosmos3-Super-Image2Video-4Step and drops the scheduler swap this
+checkpoint doesn't want -- see that file's docstring).
+
+Run in the ModelOpt venv, from the directory containing serve_cosmos3_i2v4step_diffusers.py:
+    CUDA_VISIBLE_DEVICES=0 python repackage_for_hf_i2v4step.py --format fp8 \
+        --serve-dir ./cosmos3-i2v4step-fp8-serve \
+        --out-dir   ./cosmos3-i2v4step-fp8-hf \
+        [--cache ./cosmos3-i2v4step-cache]
+
+Then verify the result loads + renders:
+    python -i load_cosmos3_modelopt.py ./cosmos3-i2v4step-fp8-hf
+"""
+import argparse
+import json
+import os
+import pathlib
+import shutil
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch  # noqa: F401  (ensures CUDA init / dtype availability)
+from modelopt.torch.opt import enable_huggingface_checkpointing
+
+from serve_cosmos3_i2v4step_diffusers import build_quantized_transformer, try_restore_quantized
+
+QUANT_TYPE = {"fp8": "FP8", "nvfp4": "NVFP4"}
+
+
+def ensure_loadable_config(transformer_dir: str, fmt: str) -> None:
+    """If save_pretrained wrote a quantization_config, make sure diffusers can construct
+    it: NVIDIAModelOptConfig needs `quant_type`, and a truthy `modelopt_config` avoids the
+    buggy get_config_from_quant_type() builder. (Structure restore itself comes from
+    modelopt_state.pth; this just keeps config parsing from crashing on load.)"""
+    cfg_path = pathlib.Path(transformer_dir) / "config.json"
+    cfg = json.loads(cfg_path.read_text())
+    qc = cfg.get("quantization_config")
+    if isinstance(qc, dict):
+        qc["quant_type"] = QUANT_TYPE[fmt]
+        qc.setdefault("weight_only", True)
+        if not qc.get("modelopt_config"):
+            qc["modelopt_config"] = {"quant_cfg": {}, "algorithm": "max"}
+        cfg["quantization_config"] = qc
+        cfg_path.write_text(json.dumps(cfg, indent=2))
+        print(f"[patch] quantization_config made loadable (quant_type={qc['quant_type']})")
+    else:
+        print("[patch] no embedded quantization_config; relying on modelopt_state.pth for restore")
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--format", choices=["fp8", "nvfp4"], default="fp8")
+    ap.add_argument("--serve-dir", required=True,
+                    help="existing assembled pipeline dir (source of VAE / tokenizers / model_index.json)")
+    ap.add_argument("--out-dir", required=True, help="new drop-in repo dir to create")
+    ap.add_argument("--cache", default=None, help="optional mto cache dir for a faster rebuild")
+    ap.add_argument("--gpu-mem-fraction", type=float, default=0.85)
+    args = ap.parse_args()
+
+    model = try_restore_quantized(args.format, args.cache) if args.cache else None
+    if model is None:
+        model = build_quantized_transformer(args.format, args.gpu_mem_fraction)
+
+    enable_huggingface_checkpointing()
+    tdir = os.path.join(args.out_dir, "transformer")
+    os.makedirs(tdir, exist_ok=True)
+    print(f"[save] writing round-trippable transformer (+ modelopt_state.pth) -> {tdir}")
+    model.save_pretrained(tdir)
+    state_file = os.path.join(tdir, "modelopt_state.pth")
+    assert os.path.isfile(state_file), (
+        f"expected {state_file} to exist -- enable_huggingface_checkpointing() must run "
+        "before save_pretrained(); without modelopt_state.pth the repo won't load in diffusers"
+    )
+    ensure_loadable_config(tdir, args.format)
+
+    print(f"[assemble] copying non-transformer components from {args.serve_dir}")
+    for name in os.listdir(args.serve_dir):
+        if name == "transformer":
+            continue
+        src = os.path.join(args.serve_dir, name)
+        dst = os.path.join(args.out_dir, name)
+        if os.path.isdir(src):
+            shutil.copytree(src, dst, dirs_exist_ok=True)
+        else:
+            shutil.copy2(src, dst)
+
+    print(f"[done] drop-in repo -> {args.out_dir}")
+    print(f"       verify:  python -i load_cosmos3_modelopt.py {args.out_dir}")
+
+
+if __name__ == "__main__":
+    main()

scheduler/scheduler_config.json

@@ -0,0 +1,28 @@
+{
+  "_class_name": "FlowMatchEulerDiscreteScheduler",
+  "_diffusers_version": "0.39.0",
+  "base_image_seq_len": 256,
+  "base_shift": 0.5,
+  "fixed_step_requires_explicit_sigmas": true,
+  "fixed_step_sampler_config": {
+    "sample_type": "sde",
+    "t_list": [
+      1.0,
+      0.9375,
+      0.8333333333333334,
+      0.625
+    ]
+  },
+  "invert_sigmas": false,
+  "max_image_seq_len": 4096,
+  "max_shift": 1.15,
+  "num_train_timesteps": 1000,
+  "shift": 1.0,
+  "shift_terminal": null,
+  "stochastic_sampling": true,
+  "time_shift_type": "exponential",
+  "use_beta_sigmas": false,
+  "use_dynamic_shifting": false,
+  "use_exponential_sigmas": false,
+  "use_karras_sigmas": false
+}

scripts/gen_video.py

@@ -0,0 +1,76 @@
+"""Minimal image-to-video generation against a vLLM-Omni endpoint (sync mode).
+
+Run from the Cosmos3-Super-Image2Video-4Step repo root:
+
+    python scripts/gen_video.py \
+        --prompt-file assets/example_prompt.json \
+        --image-path assets/example_first_frame.png \
+        --output-path scripts/output.mp4
+
+Use --endpoint http://<server-host>:8000 if the vLLM-Omni server is not local.
+"""
+
+import argparse
+import json
+from pathlib import Path
+
+import requests
+
+# Fixed generation settings: 16:9 480p, 189 frames @ 24 fps.
+ASPECT_RATIO = "16,9"
+WIDTH = 832
+HEIGHT = 480
+NUM_FRAMES = 189
+FPS = 24
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Generate one I2V sample (sync mode).")
+    parser.add_argument("--endpoint", default="http://localhost:8000", help="vLLM-Omni endpoint base URL.")
+    parser.add_argument("--prompt-file", type=Path, default=Path("assets/example_prompt.json"))
+    parser.add_argument("--image-path", type=Path, default=Path("assets/example_first_frame.png"))
+    parser.add_argument("--output-path", type=Path, default=Path("scripts/output.mp4"))
+    args = parser.parse_args()
+
+    spec = json.loads(args.prompt_file.read_text(encoding="utf-8"))
+    prompt = json.loads(spec["prompt"])
+    prompt["duration"] = f"{int(NUM_FRAMES / FPS)}s"
+    prompt["fps"] = float(round(FPS))
+    prompt["resolution"] = {"H": HEIGHT, "W": WIDTH}
+    prompt["aspect_ratio"] = ASPECT_RATIO
+
+    request_body = {
+        "prompt": json.dumps(prompt, ensure_ascii=False),
+        "negative_prompt": spec["negative_prompt"],
+        "size": f"{WIDTH}x{HEIGHT}",
+        "num_frames": NUM_FRAMES,
+        "fps": FPS,
+        "guidance_scale": 1.0,
+        "extra_params": json.dumps(
+            {
+                "guardrails": True,
+                "use_resolution_template": False,
+                "use_duration_template": False,
+            }
+        ),
+    }
+    files = {"input_reference": ("input.png", args.image_path.read_bytes(), "image/png")}
+    headers = {"Accept": "video/mp4", "User-Agent": "curl/8.5.0"}
+
+    endpoint = args.endpoint.rstrip("/")
+    response = requests.post(
+        f"{endpoint}/v1/videos/sync",
+        data=request_body,
+        files=files,
+        headers=headers,
+        timeout=(10, 600),
+    )
+    response.raise_for_status()
+
+    args.output_path.parent.mkdir(parents=True, exist_ok=True)
+    args.output_path.write_bytes(response.content)
+    print(f"Saved video to {args.output_path} ({len(response.content) / (1024 * 1024):.1f} MB)")
+
+
+if __name__ == "__main__":
+    main()

scripts/upsample_prompt.py

@@ -0,0 +1,168 @@
+"""Minimal image-to-video prompt upsampler.
+
+Run from the Cosmos3-Super-Image2Video-4Step repo root:
+
+    export PROMPT_UPSAMPLER_API_KEY="..."
+    python scripts/upsample_prompt.py \
+        --model-name <model> \
+        --base-url <VLM-endpoint-url> \
+        --image-path assets/example_first_frame.png \
+        --user-prompt "The dog flies into the outer space" \
+        --output-path scripts/upsampled.json
+"""
+
+import argparse
+import base64
+import json
+import mimetypes
+import os
+import re
+from pathlib import Path
+
+import requests
+
+# Fixed generation settings: 16:9 480p, 189 frames @ 24 fps.
+ASPECT_RATIO = "16,9"
+WIDTH = 832
+HEIGHT = 480
+NUM_FRAMES = 189
+FPS = 24
+
+MAX_TOKENS = 8192
+
+PROMPT_TEMPLATE = """You are an expert prompt engineer for an image-to-video generative model. You are given a STARTING FRAME image (the first frame of the video) and a USER INSTRUCTION describing the desired motion or changes to animate. Your task is to produce a dense, cinematic video description that the model will use to generate the full video, together with a customized negative prompt.
+
+Complete this task in two phases.
+
+---
+### PHASE 1: VIDEO DESCRIPTION
+Write a dense, narrative temporal_caption inside `<final_prompt>` XML tags, formatted as a JSON object using this exact template:
+
+{
+  "temporal_caption": "...",
+  "duration": "placeholder",
+  "fps": "placeholder",
+  "resolution": {
+    "H": "placeholder",
+    "W": "placeholder"
+  },
+  "aspect_ratio": "placeholder"
+}
+
+Rules for the temporal_caption:
+- The provided image is the exact starting frame - all described motion must be consistent with the starting frame.
+- Opening: Establish the scene — subjects, environment, lighting — describing what is directly visible in the starting frame accurately and faithfully, noting essential elements that the motion will directly involve, the subject's orientation (e.g., "facing away", "in three-quarter profile"), and any implied ongoing motion (e.g., a cyclist leaning into a curve, water already splashing) so the video continues smoothly. Phrase it naturally as a scene description (do not say "in the starting frame", "initially shown", or similar meta-references).
+- Motion: Describe the changes and actions in chronological order. Flow naturally from one action to the next. Advance time using natural conjunctions (e.g., "while," "as," "and").
+- Physical Accuracy: All motion must obey gravity and reflect realistic material behavior (e.g., cloth ripples, water splashes, rigid objects resist deformation).
+- Cause-and-effect: Always describe causes before their effects. Reflections, shadows, and secondary effects cannot appear on their own — the source object must first enter the frame or move into the relevant position before any reflection or shadow is described. E.g., a person must walk to the water's edge before their reflection appears on the surface; an object must strike the water before a splash erupts.
+- Object Permanence: Every subject must persist throughout or have a clear reason for entering or exiting. When a new subject not present in the starting frame is introduced (e.g., an opposing team, an arriving vehicle), briefly describe their appearance (e.g., uniform color, vehicle type and color) so the generator can render them consistently, and describe a logical way for them to come into the frame (e.g., entering from a specific side of the frame, walking in through a door, or emerging from behind an existing object) rather than having them appear out of nowhere.
+- Taboo Phrases: NEVER refer to the video medium itself. Avoid "the video shows...", "the scene...", "the clip...", "the frame...", "the camera shows...", "we see...".
+- Perspective: Describe human body sides from the subject's own perspective (e.g., "her right hand" = the subject's right hand) to avoid ambiguity. This applies whenever a body part enters or moves in the frame: always specify whether it is the left or right (e.g., "his right hand reaches in from the lower edge"), never a bare "a hand enters the frame".
+- Pronouns: Use singular pronouns ("he", "she", "him", "her", "it") or a singular noun phrase ("the person", "the rider", "the child") for single subjects. Never use "they"/"them"/"their" to refer to one person, as this can cause the model to render multiple subjects.
+- Spatial Phrasing: Use spatial relationships for motion (e.g., "enters from the left", "rises above the horizon") rather than camera-centric descriptions.
+- Camera: Include camera motion only if specified in the instruction; otherwise describe from a static viewpoint. Keep any described camera movement subtle and gradual — do not exaggerate altitude loss, tilt angle, or speed beyond what is minimally implied by the instruction. Do not use the word "transition" when describing camera motion.
+- Cinematography Terms: When the instruction references a lens, camera, or filming technique (e.g., "probe lens", "macro lens", "fisheye", "drone shot", "GoPro"), treat it as a cinematographic style describing how the footage is captured — never as a physical object visible in the scene. Mention the style (e.g., for a probe lens: extreme close shot; for a fisheye lens: extreme wide angle fisheye view) rather than mentioning the lens or camera apparatus itself.
+- Timelapse: If the instruction implies timelapse, explicitly use the word "timelapse" in the caption and avoid exaggerating its effects.
+- Cuts & Montages: Always describe a single continuous shot with no hard cuts unless the user instruction explicitly used words like "cut", "hard cut", "jump cut", "shot change", or "montage". When multiple shots are requested without specifying an exact number, describe at most 3 shots, and dedicate the majority of the description to the opening action before any cut. Never use phrases like "the first shot", "the opening shot", or number shots as "first", "second", etc. — simply describe the action directly.
+- Tone: Neutral, objective, descriptive. No opinions, value judgments, or inferred emotions unless physically observable.
+- Length & Format: Write exactly ONE coherent paragraph of 5-8 sentences. No bullet points or lists.
+
+USER INSTRUCTION:
+"{nl_description}"
+
+---
+### PHASE 2: NEGATIVE PROMPT
+Using your final video description from Phase 1, create a customized negative prompt.
+
+HOW IT WORKS:
+A negative prompt describes exactly what a bad video looks like. Use declarative statements (e.g., "blurry faces"). Never use negative instructions like "avoid" or "do not".
+
+---
+DEFAULT NEGATIVE PROMPT:
+The video captures a series of frames showing macroblocking artifacts, chromatic aberration, high-frequency noise, and rolling shutter distortion. It includes static with no motion, motion blur, over-saturation, shaky footage, low resolution, grainy texture, pixelated images, poorly lit areas, underexposed and overexposed scenes, poor color balance, washed out colors, choppy sequences, jerky movements, low frame rate, bit-depth compression artifacts, color banding, unnatural transitions, outdated special effects, fake elements, unconvincing visuals, poorly edited content, jump cuts, hard cut, visual noise, and flickering. It features moiré patterns, edge halos, and temporal aliasing. Furthermore, the content defies common sense, generating illogical scenarios, nonsensical entities, absurd character behaviors, and conceptual paradoxes that violate basic human reasoning and everyday reality. The video looks like a surreal or glitchy hallucination. Overall, the video is of poor quality.
+---
+
+INSTRUCTIONS:
+Delete any words from the default negative prompt that contradict your intended video. Keep most of the original wording and structure intact, and do not add new items. Examples:
+* If you want scene cuts/montages -> REMOVE "jump cuts" and "hard cut".
+* If you want a motionless/static scene -> REMOVE "static with no motion".
+* If you want fantasy, sci-fi, or surrealism -> REMOVE "defies common sense", "illogical scenarios", "nonsensical entities", "surreal", and related logic-violation terms.
+* If the scene has flickering light -> REMOVE "flickering".
+* If it is a night-time timelapse -> REMOVE "motion blur".
+
+Output only the final negative prompt as a single paragraph, wrapped in <negative_prompt> tags. Do not output any explanation or preamble."""
+
+
+def image_to_data_url(path: Path) -> str:
+    """Encode a local image as a base64 data URL."""
+    mime = mimetypes.guess_type(path.name)[0] or "image/png"
+    encoded = base64.b64encode(path.read_bytes()).decode("ascii")
+    return f"data:{mime};base64,{encoded}"
+
+
+def extract_tag(text: str, tag: str) -> str | None:
+    """Return the stripped inner text of the first <tag>...</tag> block, if present."""
+    match = re.search(rf"<{tag}>(.*?)</{tag}>", text, flags=re.DOTALL)
+    return match.group(1).strip() if match else None
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="Upsample an image-to-video prompt with a VLM.")
+    parser.add_argument("--image-path", type=Path, default=Path("assets/example_first_frame.png"))
+    parser.add_argument("--user-prompt", default="The dog flies into the outer space")
+    parser.add_argument("--output-path", type=Path, default=Path("scripts/upsampled.json"))
+    parser.add_argument("--model-name", required=True)
+    parser.add_argument("--base-url", required=True, metavar="<VLM-endpoint-url>")
+    return parser.parse_args()
+
+
+def invoke_vlm(image_path: Path, user_prompt: str, model_name: str, base_url: str) -> str:
+    """Call an OpenAI-compatible chat completions endpoint and return the assistant text."""
+    payload = {
+        "model": model_name,
+        "max_tokens": MAX_TOKENS,
+        "messages": [
+            {
+                "role": "user",
+                "content": [
+                    {"type": "image_url", "image_url": {"url": image_to_data_url(image_path)}},
+                    {"type": "text", "text": PROMPT_TEMPLATE.replace("{nl_description}", user_prompt.strip())},
+                ],
+            }
+        ],
+    }
+    headers = {"Authorization": f"Bearer {os.environ['PROMPT_UPSAMPLER_API_KEY']}"}
+    response = requests.post(f"{base_url.rstrip('/')}/chat/completions", json=payload, headers=headers)
+    response.raise_for_status()
+    return response.json()["choices"][0]["message"]["content"]
+
+
+def main() -> None:
+    args = parse_args()
+    content = invoke_vlm(args.image_path, args.user_prompt, args.model_name, args.base_url)
+
+    final_prompt = extract_tag(content, "final_prompt")
+    if final_prompt is None:
+        raise RuntimeError(f"Response missing <final_prompt> block:\n{content}")
+
+    # Pin the output parameters post-hoc (the template leaves them as placeholders).
+    data = json.loads(final_prompt)
+    data["duration"] = f"{int(NUM_FRAMES / FPS)}s"
+    data["fps"] = float(round(FPS))
+    data["resolution"] = {"H": HEIGHT, "W": WIDTH}
+    data["aspect_ratio"] = ASPECT_RATIO
+
+    record: dict = {"prompt": json.dumps(data, ensure_ascii=False)}
+    negative = extract_tag(content, "negative_prompt")
+    if negative:
+        record["negative_prompt"] = negative
+
+    args.output_path.parent.mkdir(parents=True, exist_ok=True)
+    args.output_path.write_text(json.dumps(record, ensure_ascii=False), encoding="utf-8")
+    print(f"PROMPT:\n{record['prompt']}")
+    print(f"\nNEGATIVE PROMPT:\n{record.get('negative_prompt', '')}")
+    print(f"\nWrote {args.output_path}")
+
+
+if __name__ == "__main__":
+    main()

serve_cosmos3_i2v4step_diffusers.py

@@ -0,0 +1,449 @@
+#!/usr/bin/env python
+"""
+Local generation server for your quantized Cosmos3-Super-Image2Video-4Step, built on
+the *validated* diffusers path (NOT vLLM-Omni). Sibling of serve_cosmos3_diffusers.py
+(the nvidia/Cosmos3-Super base-model server) -- same build recipe, adapted for three
+real differences in this checkpoint:
+
+  1. NO /generate (text -> still) ENDPOINT. This checkpoint is Image2Video-specific
+     (unlike the T2I/T2V/I2V-omni base model); NVIDIA's docs never demonstrate a
+     text-only call for it, and there's no local way to verify one works without
+     the checkpoint in hand. Rather than ship a silently-untested endpoint, this
+     server exposes /animate only. If a text-only path turns out to work when you
+     test it, port /generate over from serve_cosmos3_diffusers.py at that point.
+  2. NO SCHEDULER OBJECT SWAP, BUT A PATCHED set_timesteps. serve_cosmos3_diffusers.py
+     replaces the shipped scheduler with UniPCMultistepScheduler(flow_shift=...); this
+     checkpoint instead keeps FlowMatchEulerDiscreteScheduler as shipped (do not swap
+     it) but needs one targeted patch: Cosmos3OmniPipeline.__call__ always calls
+     scheduler.set_timesteps(num_inference_steps, device=device) with no passthrough
+     for this checkpoint's scheduler_config.json fixed_step_sampler_config (verified by
+     running it -- an unpatched call silently executed the pipeline's 35-step default
+     instead of the checkpoint's trained 4-step sde schedule). make_pipeline() below
+     applies _force_fixed_step_schedule() to fix this.
+  3. NO num_inference_steps IN THE REQUEST; guidance_scale IS FIXED AT 1.0. Per
+     NVIDIA's model card these are fixed by the distilled checkpoint -- CFG is
+     distilled out (do_classifier_free_guidance is guidance_scale != 1.0 in the
+     pipeline source, so the 6.0 pipeline default would silently re-enable it) and
+     the step count comes from the scheduler patch above, not from the caller.
+
+ENDPOINTS
+---------
+  GET  /health    -> readiness + which format is loaded
+  POST /animate   -> image -> video (multipart upload; returns MP4, or GIF if no
+                     mp4 encoder is installed)
+
+USAGE
+-----
+    CUDA_VISIBLE_DEVICES=0 python serve_cosmos3_i2v4step_diffusers.py --repo prometheusAIR/Cosmos3-I2V4Step-fp8
+    CUDA_VISIBLE_DEVICES=0 python serve_cosmos3_i2v4step_diffusers.py --repo ./cosmos3-i2v4step-fp8-hf
+
+    # or rebuild from the bf16 source (the original streaming path):
+    CUDA_VISIBLE_DEVICES=0 python serve_cosmos3_i2v4step_diffusers.py --format fp8
+    CUDA_VISIBLE_DEVICES=0 python serve_cosmos3_i2v4step_diffusers.py --format fp8 --cache ./cosmos3-i2v4step-cache
+
+Image -> video:
+    curl -s -X POST http://localhost:8000/animate \
+        -F image=@out.png \
+        -F 'prompt=The robotic arm slowly lowers its gripper toward the objects and holds. Static camera.' \
+        -F num_frames=49 -F fps=24 \
+        --output clip.mp4
+
+Health:
+    curl -s http://localhost:8000/health
+"""
+
+import argparse
+import asyncio
+import contextlib
+import gc
+import io
+import os
+import tempfile
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch
+from accelerate import init_empty_weights, load_checkpoint_in_model
+from accelerate.utils import get_max_memory, infer_auto_device_map
+from accelerate.utils.dataclasses import CustomDtype
+from fastapi import FastAPI, File, Form, UploadFile
+from fastapi.responses import Response
+from huggingface_hub import snapshot_download
+from PIL import Image
+from pydantic import BaseModel
+
+import modelopt.torch.quantization as mtq
+from diffusers import Cosmos3OmniTransformer
+from diffusers.utils import export_to_gif, export_to_video
+
+SRC_REPO = "nvidia/Cosmos3-Super-Image2Video-4Step"
+
+SPARE_SUBSTRINGS = [
+    "time_embedder", "proj_in", "proj_out", "lm_head", "embed", "norm", "audio_proj",
+]
+
+
+def _is_spare(name: str) -> bool:
+    return any(s in name for s in SPARE_SUBSTRINGS)
+
+
+def build_quant_cfg(fmt: str) -> dict:
+    if fmt == "fp8":
+        return {
+            "quant_cfg": {
+                "*weight_quantizer": {"num_bits": (4, 3), "axis": None, "enable": True},
+                "*input_quantizer": {"enable": False},
+                "*output_quantizer": {"enable": False},
+                "*softmax_quantizer": {"enable": False},
+            },
+            "algorithm": "max",
+        }
+    if fmt == "nvfp4":
+        import copy
+        # Return the base preset UNMODIFIED -- on this installed modelopt version,
+        # W4A16_NVFP4_CFG's quant_cfg is a LIST (newer format), not the dict this
+        # function used to assume (confirmed empirically: "TypeError: list indices
+        # must be integers or slices, not str" trying dict-style key assignment
+        # here). enforce_weight_only_and_spare() below disables activations/spares
+        # on the model's actual inserted quantizer modules AFTER mtq.quantize(),
+        # which is config-form agnostic -- no need to pre-bake into the config dict.
+        # load_cosmos3_modelopt.py's loader re-applies the same disabling on every
+        # restore anyway (modelopt_state replays the config, not our imperative
+        # .disable() calls), so skipping the pre-bake here loses nothing.
+        base = getattr(mtq, "W4A16_NVFP4_CFG", None) or mtq.NVFP4_DEFAULT_CFG
+        return copy.deepcopy(base)
+    raise ValueError(f"Unknown format: {fmt!r}")
+
+
+def enforce_weight_only_and_spare(model) -> tuple[int, int]:
+    n_spare = n_act = 0
+    for name, module in model.named_modules():
+        if not (name.endswith("_quantizer") and hasattr(module, "disable")):
+            continue
+        if name.endswith("weight_quantizer"):
+            if _is_spare(name.rsplit(".", 1)[0]):
+                module.disable()
+                n_spare += 1
+        else:
+            module.disable()
+            n_act += 1
+    return n_spare, n_act
+
+
+def compressed_device_map(model, gpu_mem_fraction: float = 0.85) -> dict:
+    max_memory = {k: v * gpu_mem_fraction for k, v in get_max_memory().items()}
+    no_split = set()
+    for name, module in model.named_modules():
+        if name.endswith((".layers.0", ".blocks.0", ".transformer_blocks.0")):
+            no_split.add(module.__class__.__name__)
+    special_dtypes = {}
+    for name, module in model.named_modules():
+        if (
+            hasattr(module, "weight")
+            and hasattr(module, "weight_quantizer")
+            and getattr(module.weight_quantizer, "is_enabled", True)
+            and not getattr(module.weight_quantizer, "fake_quant", True)
+        ):
+            nb = module.weight_quantizer.num_bits
+            if isinstance(nb, tuple):
+                nb = nb[0] + nb[1] + 1
+            special_dtypes[name + ".weight"] = CustomDtype.FP8 if nb == 8 else CustomDtype.INT4
+    return infer_auto_device_map(
+        model, max_memory=max_memory,
+        no_split_module_classes=list(no_split), special_dtypes=special_dtypes,
+    )
+
+
+def _materialize_residual_meta(model) -> int:
+    n = 0
+    for _, module in model.named_modules():
+        for bn, buf in list(module._buffers.items()):
+            if buf is not None and getattr(buf, "is_meta", False):
+                module._buffers[bn] = torch.zeros(buf.shape, dtype=buf.dtype, device="cuda")
+                n += 1
+        for pn, par in list(module._parameters.items()):
+            if par is not None and getattr(par, "is_meta", False):
+                module._parameters[pn] = torch.nn.Parameter(
+                    torch.zeros(par.shape, dtype=par.dtype, device="cuda"), requires_grad=False
+                )
+                n += 1
+    return n
+
+
+def _transformer_dir() -> str:
+    local_root = snapshot_download(SRC_REPO, allow_patterns=["transformer/*"])
+    return os.path.join(local_root, "transformer")
+
+
+def build_quantized_transformer(fmt: str, gpu_mem_fraction: float = 0.85):
+    """The proven path: empty-on-meta -> quantize -> compress -> stream shards in."""
+    transformer_dir = _transformer_dir()
+    print(f"[build] empty transformer on meta from {transformer_dir}")
+    config = Cosmos3OmniTransformer.load_config(transformer_dir)
+    with init_empty_weights(include_buffers=False):
+        model = Cosmos3OmniTransformer.from_config(config)
+
+    print(f"[build] inserting weight-only {fmt} quantizers")
+    mtq.quantize(model, build_quant_cfg(fmt))
+    n_spare, n_act = enforce_weight_only_and_spare(model)
+    print(f"[build] weight-only: disabled {n_act} activation quantizers; {n_spare} spare weight layers")
+
+    print("[build] setting up compressed parameter shapes")
+    try:
+        mtq.compress(model, config=mtq.CompressConfig(quant_gemm=False))
+    except (AttributeError, TypeError):
+        mtq.compress(model)
+
+    print("[build] streaming BF16 shards into compressed form (slow step)")
+    load_checkpoint_in_model(
+        model, checkpoint=transformer_dir,
+        device_map=compressed_device_map(model, gpu_mem_fraction), dtype=torch.bfloat16,
+    )
+    fixed = _materialize_residual_meta(model)
+    if fixed:
+        print(f"[build] materialized {fixed} residual meta tensors")
+    return model
+
+
+# --- optional fast-restart cache (modelopt_state + weights, per ModelOpt docs) ----------
+def _cache_paths(cache_dir: str, fmt: str):
+    return (os.path.join(cache_dir, f"modelopt_state_{fmt}.pt"),
+            os.path.join(cache_dir, f"weights_{fmt}.pt"))
+
+
+def save_quantized(model, fmt: str, cache_dir: str) -> None:
+    try:
+        from modelopt.torch.opt import modelopt_state
+    except ImportError:
+        from modelopt.torch.opt.conversion import modelopt_state
+    os.makedirs(cache_dir, exist_ok=True)
+    state_path, weights_path = _cache_paths(cache_dir, fmt)
+    print(f"[cache] writing {state_path} + {weights_path} (large; one time)")
+    torch.save(modelopt_state(model), state_path)
+    torch.save(model.state_dict(), weights_path)
+
+
+def try_restore_quantized(fmt: str, cache_dir: str):
+    state_path, weights_path = _cache_paths(cache_dir, fmt)
+    if not (os.path.isfile(state_path) and os.path.isfile(weights_path)):
+        return None
+    try:
+        try:
+            from modelopt.torch.opt import restore_from_modelopt_state
+        except ImportError:
+            from modelopt.torch.opt.conversion import restore_from_modelopt_state
+        print(f"[cache] restoring from {state_path}")
+        config = Cosmos3OmniTransformer.load_config(_transformer_dir())
+        with init_empty_weights(include_buffers=False):
+            model = Cosmos3OmniTransformer.from_config(config)
+        state = torch.load(state_path, map_location="cpu", weights_only=False)
+        restore_from_modelopt_state(model, state)
+        weights = torch.load(weights_path, map_location="cpu", weights_only=False)
+        model.load_state_dict(weights, strict=False, assign=True)
+        _materialize_residual_meta(model)
+        print("[cache] restore OK")
+        return model
+    except Exception as e:
+        import traceback
+        print(f"[cache] restore failed ({type(e).__name__}: {e}); falling back to full rebuild")
+        traceback.print_exc()
+        return None
+
+
+def _force_fixed_step_schedule(scheduler) -> bool:
+    """See module docstring point 2. Patches THIS scheduler instance's set_timesteps
+    to always use the checkpoint's own fixed_step_sampler_config.t_list, regardless of
+    whatever num_inference_steps Cosmos3OmniPipeline.__call__ passes in internally.
+
+    ALSO disables stochastic_sampling (forces the deterministic ODE branch) -- CONFIRMED
+    by direct A/B render (same seed/image/prompt), not speculative. This checkpoint's
+    scheduler ships stochastic_sampling=True (SDE). Cosmos3's image-conditioning anchors
+    frame 0 by zeroing the model's predicted velocity there; that only means "leave this
+    position unchanged" under the deterministic step (prev_sample = sample + dt*0). The
+    SDE branch instead computes x0 = sample - current_sigma*model_output (= sample when
+    velocity is 0), then prev_sample = (1-next_sigma)*x0 + next_sigma*randn_tensor(...) --
+    it re-noises by next_sigma regardless of velocity, with no zero-velocity no-op. Over
+    this checkpoint's 4 steps that compounds to ~99.6% fresh noise in the conditioned
+    frame by the end: the input image comes out as colorful static while the genuinely
+    denoised motion frames still look like a plausible video. Disabling stochastic_
+    sampling restores the correct zero-velocity-is-a-no-op behavior."""
+    cfg = getattr(scheduler.config, "fixed_step_sampler_config", None)
+    t_list = cfg.get("t_list") if isinstance(cfg, dict) else None
+    if not t_list:
+        print("[warn] no fixed_step_sampler_config.t_list on this scheduler; leaving set_timesteps unpatched")
+        return False
+    _orig = scheduler.set_timesteps
+
+    def _patched(num_inference_steps=None, device=None, sigmas=None, mu=None, timesteps=None):
+        return _orig(sigmas=list(t_list), device=device)
+
+    scheduler.set_timesteps = _patched
+    if getattr(scheduler.config, "stochastic_sampling", False):
+        scheduler.register_to_config(stochastic_sampling=False)
+        print("[build] disabled stochastic_sampling (SDE re-noising corrupts the "
+              "image-conditioned frame -- see docstring)")
+    print(f"[build] forced fixed {len(t_list)}-step sde schedule: {t_list}")
+    return True
+
+
+# --- pipeline assembly --------------------------------------------------------------
+def make_pipeline(model):
+    from diffusers import Cosmos3OmniPipeline
+
+    model.to("cuda")
+    for m in model.modules():
+        for bn, buf in list(m._buffers.items()):
+            if buf is not None and buf.dtype == torch.float32:
+                m._buffers[bn] = buf.to(torch.bfloat16)
+
+    def _cast_bf16(_m, args):
+        return tuple(
+            a.to(torch.bfloat16)
+            if torch.is_tensor(a) and a.is_floating_point() and a.dtype != torch.bfloat16 else a
+            for a in args
+        )
+
+    for name, m in model.named_modules():
+        if "time_embedder" in name and hasattr(m, "linear_1"):
+            m.register_forward_pre_hook(_cast_bf16)
+
+    pipe = Cosmos3OmniPipeline.from_pretrained(
+        SRC_REPO, transformer=model, torch_dtype=torch.bfloat16,
+        enable_safety_checker=False,  # local single-user server; revisit if exposing it
+    )
+    # Keep the shipped FlowMatchEulerDiscreteScheduler object, but patch its
+    # set_timesteps to actually honor the checkpoint's fixed 4-step sde schedule.
+    _force_fixed_step_schedule(pipe.scheduler)
+    for name, comp in pipe.components.items():
+        if name != "transformer" and isinstance(comp, torch.nn.Module):
+            comp.to("cuda")
+    return pipe
+
+
+# --- HTTP server ------------------------------------------------------------------------
+STATE: dict = {}
+_gen_lock = asyncio.Lock()  # one generation at a time on a single GPU
+
+
+def _run_i2v(pil_image, prompt, negative_prompt, num_frames, fps, height, width, seed) -> tuple[bytes, str]:
+    pipe = STATE["pipe"]
+    image = pil_image.convert("RGB")  # the pipeline resizes this to (height, width)
+    gen = torch.Generator(device="cuda").manual_seed(int(seed)) if seed >= 0 else None
+    with torch.inference_mode():
+        # num_inference_steps is a no-op once _force_fixed_step_schedule has patched
+        # the scheduler (it always substitutes the checkpoint's own t_list).
+        # guidance_scale=1.0 disables CFG -- see module docstring point 3.
+        result = pipe(
+            prompt=prompt, negative_prompt=negative_prompt,
+            image=image, num_frames=num_frames, fps=fps,
+            height=height, width=width,
+            num_inference_steps=4, guidance_scale=1.0,
+            generator=gen, output_type="pil",
+        )
+    frames = result.video
+    try:
+        with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tf:
+            path = tf.name
+        export_to_video(frames, path, fps=int(round(fps)))
+        media = "video/mp4"
+    except Exception:  # no mp4 backend installed -> GIF (PIL-only, always works)
+        with tempfile.NamedTemporaryFile(suffix=".gif", delete=False) as tf:
+            path = tf.name
+        export_to_gif(frames, path)
+        media = "image/gif"
+    data = open(path, "rb").read()
+    os.remove(path)
+    del result
+    gc.collect()
+    torch.cuda.empty_cache()
+    return data, media
+
+
+@contextlib.asynccontextmanager
+async def lifespan(app: FastAPI):
+    fmt = STATE["fmt"]
+    repo = STATE.get("repo")
+    if repo:
+        from load_cosmos3_modelopt import load_pipe
+
+        path = repo
+        if not os.path.isdir(path):
+            from huggingface_hub import snapshot_download as _snap
+            print(f"[serve] fetching full snapshot of {repo} from the Hub ...")
+            path = _snap(repo)
+        print(f"[serve] loading drop-in checkpoint: {path}")
+        pipe = load_pipe(path)
+        # load_cosmos3_modelopt.load_pipe() is shared with the base model and doesn't
+        # know about this checkpoint's fixed-step schedule -- patch it here instead.
+        _force_fixed_step_schedule(pipe.scheduler)
+        STATE["pipe"] = pipe
+        print(f"[serve] ready (drop-in: {repo}, {fmt.upper()} assumed from repo contents)")
+        yield
+        STATE.clear()
+        return
+    cache_dir = STATE.get("cache_dir")
+    model = None
+    if cache_dir:
+        model = try_restore_quantized(fmt, cache_dir)
+    if model is None:
+        model = build_quantized_transformer(fmt, STATE["gpu_mem_fraction"])
+        if cache_dir:
+            try:
+                save_quantized(model, fmt, cache_dir)
+            except Exception as e:
+                print(f"[cache] save failed ({type(e).__name__}: {e}); continuing without cache")
+    STATE["pipe"] = make_pipeline(model)
+    print(f"[ready] serving {fmt.upper()} Cosmos3-Super-Image2Video-4Step on diffusers")
+    yield
+    STATE.clear()
+
+
+app = FastAPI(lifespan=lifespan)
+
+
+@app.get("/health")
+async def health():
+    return {"status": "ok" if "pipe" in STATE else "loading", "format": STATE.get("fmt")}
+
+
+@app.post("/animate")
+async def animate(
+    image: UploadFile = File(...),
+    prompt: str = Form(...),
+    negative_prompt: str = Form(""),
+    num_frames: int = Form(49),         # ~2.04s @ 24fps; 4n+1 maps cleanly to the VAE's 4x temporal compression
+    fps: float = Form(24.0),
+    height: int = Form(1024),
+    width: int = Form(1024),
+    seed: int = Form(1234),             # pass -1 for a random clip each call
+):
+    pil = Image.open(io.BytesIO(await image.read()))
+    async with _gen_lock:
+        loop = asyncio.get_running_loop()
+        data, media = await loop.run_in_executor(
+            None, _run_i2v, pil, prompt, negative_prompt, num_frames, fps, height, width, seed,
+        )
+    return Response(content=data, media_type=media)
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--format", choices=["fp8", "nvfp4"], default="fp8")
+    ap.add_argument("--repo", default=None,
+                    help="Serve a published drop-in checkpoint: an HF repo id or a local "
+                         "repackaged dir. Skips the bf16 rebuild entirely; requires "
+                         "load_cosmos3_modelopt.py next to this file.")
+    ap.add_argument("--host", default="0.0.0.0")
+    ap.add_argument("--port", type=int, default=8000)
+    ap.add_argument("--gpu-mem-fraction", type=float, default=0.85)
+    ap.add_argument("--cache", default=None,
+                    help="Dir for a fast-restart cache. First boot rebuilds + writes it; "
+                         "later boots restore from it. Any restore error -> full rebuild.")
+    args = ap.parse_args()
+
+    STATE.update(
+        fmt=args.format, gpu_mem_fraction=args.gpu_mem_fraction, cache_dir=args.cache, repo=args.repo,
+    )
+
+    import uvicorn
+    uvicorn.run(app, host=args.host, port=args.port)

text_tokenizer/added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

text_tokenizer/chat_template.jinja

@@ -0,0 +1,120 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' }}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- for message in messages %}
+    {%- if message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content_item in message.content %}
+                {%- if 'text' in content_item %}
+                    {{- content_item.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and message.content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

text_tokenizer/special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

text_tokenizer/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aeb13307a71acd8fe81861d94ad54ab689df773318809eed3cbe794b4492dae4
+size 11422654

text_tokenizer/tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 262144,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

tokenizer_config.json

@@ -0,0 +1,239 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' }}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- for message in messages %}\n    {%- if message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|vision_start|><|image_pad|><|vision_end|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|vision_start|><|video_pad|><|vision_end|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content_item in message.content %}\n                {%- if 'text' in content_item %}\n                    {{- content_item.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and message.content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|vision_start|><|image_pad|><|vision_end|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|vision_start|><|video_pad|><|vision_end|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 262144,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

transformer/config.json

@@ -0,0 +1,43 @@
+{
+  "_class_name": "Cosmos3OmniTransformer",
+  "_diffusers_version": "0.39.0.dev0",
+  "action_dim": 32,
+  "action_gen": false,
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "base_fps": 16,
+  "enable_fps_modulation": true,
+  "head_dim": 128,
+  "hidden_size": 5120,
+  "intermediate_size": 25600,
+  "latent_channel": 48,
+  "latent_patch_size": 2,
+  "num_attention_heads": 64,
+  "num_embodiment_domains": 32,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "patch_latent_dim": 192,
+  "rms_norm_eps": 1e-06,
+  "rope_axes_dim": [
+    24,
+    20,
+    20
+  ],
+  "rope_scaling": {
+    "mrope_interleaved": true,
+    "mrope_section": [
+      24,
+      20,
+      20
+    ],
+    "rope_type": "default"
+  },
+  "rope_theta": 5000000,
+  "sound_dim": null,
+  "sound_gen": false,
+  "sound_latent_fps": 25,
+  "timestep_scale": 0.001,
+  "unified_3d_mrope_reset_spatial_ids": true,
+  "unified_3d_mrope_temporal_modality_margin": 15000,
+  "vocab_size": 151936
+}

transformer/diffusion_pytorch_model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e2aa2e718722509028b354db2f24616f8e10a9f7c2782db7e2f3eecc9bce490
+size 9964515442

transformer/diffusion_pytorch_model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98d10b08cd826c462d0c6de64a3640648ea03841af48abdac24b10405ed7caf5
+size 9948271838

transformer/diffusion_pytorch_model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a417ced6bebae39de4df3eaa58f744630d46e7715e9773d34ec00787dae2eca
+size 9948271838

transformer/diffusion_pytorch_model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9416e9e32370e3e34ff2281ea90d965a3d54cc3e5956eb2abd0e0210cab8eb2
+size 8419116030

transformer/modelopt_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d622a0d180c70ff7396db6e5abafd855072d8080fc9e66bdea2eb6489d0e1560
+size 1295319

vae/config.json

@@ -0,0 +1,129 @@
+{
+  "_class_name": "AutoencoderKLWan",
+  "_diffusers_version": "0.39.0",
+  "_name_or_path": "Wan-AI/Wan2.2-TI2V-5B-Diffusers",
+  "attn_scales": [],
+  "base_dim": 160,
+  "clip_output": false,
+  "decoder_base_dim": 256,
+  "dim_mult": [
+    1,
+    2,
+    4,
+    4
+  ],
+  "dropout": 0.0,
+  "in_channels": 12,
+  "is_residual": true,
+  "latents_mean": [
+    -0.2289,
+    -0.0052,
+    -0.1323,
+    -0.2339,
+    -0.2799,
+    0.0174,
+    0.1838,
+    0.1557,
+    -0.1382,
+    0.0542,
+    0.2813,
+    0.0891,
+    0.157,
+    -0.0098,
+    0.0375,
+    -0.1825,
+    -0.2246,
+    -0.1207,
+    -0.0698,
+    0.5109,
+    0.2665,
+    -0.2108,
+    -0.2158,
+    0.2502,
+    -0.2055,
+    -0.0322,
+    0.1109,
+    0.1567,
+    -0.0729,
+    0.0899,
+    -0.2799,
+    -0.123,
+    -0.0313,
+    -0.1649,
+    0.0117,
+    0.0723,
+    -0.2839,
+    -0.2083,
+    -0.052,
+    0.3748,
+    0.0152,
+    0.1957,
+    0.1433,
+    -0.2944,
+    0.3573,
+    -0.0548,
+    -0.1681,
+    -0.0667
+  ],
+  "latents_std": [
+    0.4765,
+    1.0364,
+    0.4514,
+    1.1677,
+    0.5313,
+    0.499,
+    0.4818,
+    0.5013,
+    0.8158,
+    1.0344,
+    0.5894,
+    1.0901,
+    0.6885,
+    0.6165,
+    0.8454,
+    0.4978,
+    0.5759,
+    0.3523,
+    0.7135,
+    0.6804,
+    0.5833,
+    1.4146,
+    0.8986,
+    0.5659,
+    0.7069,
+    0.5338,
+    0.4889,
+    0.4917,
+    0.4069,
+    0.4999,
+    0.6866,
+    0.4093,
+    0.5709,
+    0.6065,
+    0.6415,
+    0.4944,
+    0.5726,
+    1.2042,
+    0.5458,
+    1.6887,
+    0.3971,
+    1.06,
+    0.3943,
+    0.5537,
+    0.5444,
+    0.4089,
+    0.7468,
+    0.7744
+  ],
+  "num_res_blocks": 2,
+  "out_channels": 12,
+  "patch_size": 2,
+  "scale_factor_spatial": 16,
+  "scale_factor_temporal": 4,
+  "temperal_downsample": [
+    false,
+    true,
+    true
+  ],
+  "z_dim": 48
+}

vae/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:230496cb59ff85bc9c040487737c4062480cb61c71e697b197b4c30142f2a0da
+size 1409400600

validate_cosmos3_i2v4step_fp8.py

@@ -0,0 +1,77 @@
+#!/usr/bin/env python
+"""
+Standalone re-validation of the Cosmos3-Super-Image2Video-4Step FP8 quantization
+(quantize_cosmos3_i2v4step_streaming.py). Exists because the first --smoke run's
+render was invalid: it used the pipeline's default 35-step/guidance=6.0 schedule
+instead of this checkpoint's trained 4-step sde schedule (a diffusers version gap --
+Cosmos3OmniPipeline doesn't pass the scheduler_config.json's fixed_step_sampler_config
+through to scheduler.set_timesteps; see _force_fixed_step_schedule in
+serve_cosmos3_i2v4step_diffusers.py for the full rationale).
+
+NOTE: export_hf_checkpoint() (what the quantize script wrote to ./cosmos3-i2v4step-fp8)
+is NVIDIA's *deployment* format for vLLM/TRT-LLM -- it has no modelopt_state.pth, so
+diffusers' from_pretrained / load_cosmos3_modelopt.py can't load it directly (same gap
+repackage_for_hf.py exists to solve for the base model). Rather than write a second
+~65GB copy to disk just to validate, this rebuilds the quantized transformer in memory
+via the same build_quantized_transformer() used during quantization (re-streams the
+121GB source once) and renders straight from that -- no extra disk footprint beyond the
+output clip. If you want a real servable drop-in repo afterward, run
+repackage_for_hf_i2v4step.py (that DOES need the disk headroom).
+
+USAGE
+    python validate_cosmos3_i2v4step_fp8.py --format fp8 --image out.png
+"""
+import argparse
+import os
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch
+from PIL import Image
+
+from serve_cosmos3_i2v4step_diffusers import build_quantized_transformer, make_pipeline
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--format", choices=["fp8", "nvfp4"], default="fp8")
+    ap.add_argument("--gpu-mem-fraction", type=float, default=0.85)
+    ap.add_argument("--image", default="out.png")
+    ap.add_argument("--num-frames", type=int, default=49)
+    ap.add_argument("--height", type=int, default=512)
+    ap.add_argument("--width", type=int, default=512)
+    ap.add_argument("--seed", type=int, default=1234)
+    ap.add_argument("--out", default="cosmos3_i2v4step_fp8_validate.mp4")
+    ap.add_argument(
+        "--prompt",
+        default="The camera holds static as the scene continues naturally, with subtle motion and realistic physics.",
+    )
+    args = ap.parse_args()
+
+    print(f"[build] rebuilding quantized ({args.format}) transformer in memory ...")
+    model = build_quantized_transformer(args.format, args.gpu_mem_fraction)
+    pipe = make_pipeline(model)  # includes _force_fixed_step_schedule on pipe.scheduler
+
+    cond_image = Image.open(args.image).convert("RGB")
+
+    print(f"[render] {args.num_frames} frames @ {args.height}x{args.width}, seed={args.seed}")
+    with torch.inference_mode():
+        result = pipe(
+            prompt=args.prompt,
+            negative_prompt="",
+            image=cond_image,
+            num_frames=args.num_frames,
+            height=args.height,
+            width=args.width,
+            num_inference_steps=4,   # no-op once patched; kept for clarity
+            guidance_scale=1.0,       # CFG is distilled out of this checkpoint
+            generator=torch.Generator(device="cuda").manual_seed(args.seed),
+        )
+
+    from diffusers.utils import export_to_video
+    export_to_video(result.video, args.out, fps=24)
+    print(f"[done] wrote {args.out}")
+
+
+if __name__ == "__main__":
+    main()

video_preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 25165824,
+        "shortest_edge": 4096
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "video_processor_type": "Qwen3VLVideoProcessor"
+}