Upload 73 files

.gitattributes

@@ -33,3 +33,11 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+text_tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+images/benchmark-overall.png filter=lfs diff=lfs merge=lfs -text
+images/benchmark-reasoning.png filter=lfs diff=lfs merge=lfs -text
+images/benchmark-visual-audio.png filter=lfs diff=lfs merge=lfs -text
+images/benchmark-action-1.png filter=lfs diff=lfs merge=lfs -text
+assets/*.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/*.jpg filter=lfs diff=lfs merge=lfs -text
+assets/*.png filter=lfs diff=lfs merge=lfs -text

BIAS.md

@@ -0,0 +1,11 @@
+## Bias
+
+| Field | Response |
+| :---- | :---- |
+| Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing | None. |
+| Measures taken to mitigate against unwanted bias | Training, evaluation, and testing data are curated before release to filter restricted content, including content relating to protected classes. Model behavior is evaluated across Physical AI domains — robotics, autonomous vehicles, human-centric scenes, common scenes, industry, miscellaneous, and physics-oriented benchmarks — with attention to coverage across diverse demographic and contextual characteristics that affect protected-class outcomes. |
+| Which characteristic (feature) show(s) the greatest difference in performance?: | Greatest performance differences are observed in tasks requiring long-horizon temporal consistency, fine-grained physical interactions, and embodiment-specific action generation. Performance is generally stronger on common visual reasoning and world-generation tasks than on complex multi-agent, robotics-control, or tightly synchronized multimodal generation scenarios. |
+| Which feature(s) have the worst performance overall? | Performance is generally weakest in tasks requiring long-horizon temporal consistency, precise physical interactions, embodiment-specific action control, and strict audio-visual synchronization. |
+| If using internal data, description of methods implemented in data acquisition or processing, if any, to address the prevalence of identifiable biases in the training, testing, and validation data: | Bias-specific methods applied during data processing include person-presence screening, demographic-taxonomy classification (age, gender, ethnicity), embedding-based diversity analysis, and dataset balancing across sources. Internal analysis surfaced: non-person scenes are more prevalent than person-centric content; demographic-taxonomy outputs on person-present samples are most frequently "uncertain" across age, gender, and ethnicity dimensions; and source-type variation, with people-centric image and video datasets showing higher demographic signal than document-, object-, robotics-, or scene-focused datasets. *(Quantitative details in the row below.)* Downstream deployments should add bias audits, fairness evaluation, red-teaming, demographically balanced fine-tuning, or counterfactual augmentation as mitigations. |
+| Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | Dataset analytics pipelines, metadata distribution analysis, heuristic quality checks, embedding-based clustering, model-assisted filtering systems, and benchmark evaluation suites are used to assess statistical imbalances and identify patterns that may introduce bias into model behavior. |
+| Tools used to assess statistical imbalances and highlight patterns that may introduce bias into AI models: | These datasets, such as OpenImages-derived detection-to-NLP datasets, visual grounding and VQA datasets, document/image understanding datasets, video/action understanding datasets, and NVIDIA-created or curated visual datasets, do not collectively or exhaustively represent all demographic groups (and proportionally therein). For instance, automated person-presence screening did not identify a person in approximately 58% of visual samples analyzed across approximately 400 datasets, while person-present signals were identified in approximately 42% of analyzed samples. In the subset where person-present signals were identified, these datasets contain uneven representation splits across the measured visual taxonomies: age outputs were most frequently uncertain, followed by child and adult; gender outputs were most frequently uncertain, followed by male and female; and ethnicity outputs were most frequently uncertain, followed by Hispanic and White as the most frequent identified categories. Dataset-level results vary by source type, with people-centric image and video datasets containing higher person-present and demographic-taxonomy signals than document-, object-, robotics-, or scene-focused datasets. To mitigate these imbalances, we recommend considering evaluation techniques such as bias audits, task-specific fairness evaluation, and red-teaming, along with fine-tuning with demographically balanced datasets and counterfactual data augmentation to align with the desired model behavior. This evaluation used a baseline of 200 samples across all datasets, with larger subsets of up to 3,000 samples utilized for certain in-depth analyses, identified as optimal thresholds for maximizing embedder accuracy. |

EXPLAINABILITY.md

@@ -0,0 +1,16 @@
+## Explainability
+
+| Field | Response |
+| :---- | :---- |
+| Intended Application & Domain | World reasoning and generation for Physical AI. |
+| Model Type | Mixture-of-Transformers architecture with two towers. One is an autoregressive model for Physical AI reasoning; the other is a diffusion model for Physical AI generation. |
+| Intended Users | Physical AI developers, researchers, and practitioners building or evaluating autonomous vehicle, robotics, and world-generation workflows. |
+| Output | Images, videos, audio, and action commands. |
+| Tools used to evaluate datasets to identify synthetic data and ensure data authenticity. | Dataset provenance analysis, metadata validation, watermark and artifact detection, embedding-based clustering, heuristic quality checks, and model-assisted data validation pipelines are used to identify synthetic content patterns, assess dataset authenticity, and improve data quality during dataset curation. |
+| Describe how the model works | Cosmos3 is an Omni world foundation model that generates texts, images, videos, audio, and action commands from combinations of text, images, videos, and action trajectory inputs. Input tokens from multiple modalities are packed into a shared sequence and processed by our mixture-of-transformer backbone with modality-specific output heads. |
+| Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of: | None. |
+| Technical Limitations | The model may not follow text, image, video, audio, or action trajectory inputs accurately in challenging cases, especially where the input contains complex scene composition, unusual camera motion, multiple interacting agents, low lighting, high motion blur, or fine-grained physical interactions. Generated outputs may contain temporal inconsistency, object morphing, inaccurate 3D structure, or implausible physical dynamics. Generated audio may not accurately render intelligible speech, or maintain strict temporal and semantic alignment with the visual context. |
+| Verified to have met prescribed NVIDIA quality standards | Yes. |
+| Performance Metrics | Video generation is measured using PAIBench-G, RBench, PhysicsIQ, and Artifical Analysis Image2Video benchmark. Image generation uses UniGenBench and Artifical Analysis Text2Image benchmark. For transfer evaluation, we use PAIBench-C and AVBench-C. Audio generation uses internal benchmarks. Action prediction uses metrics such as action MSE, Absolute Translation Error, Relative Translation Error, Relative Rotation Error, PSNR, and robotic task completion success rate. |
+| Potential Known Risks | This model can generate synthetic media and may produce content that is offensive, unsafe, misleading, indecent, or unsuitable for a target deployment. Users should implement robust safety guardrails — including content filtering, abuse monitoring, and access controls — to reduce the risk of harmful outputs. Users are responsible for ensuring that their use of the model complies with all applicable laws and regulations, and for regularly reviewing and updating their guardrails as risks evolve. |
+| Licensing | [OpenMDW1.1](https://openmdw.ai/)  |

LICENSE

@@ -0,0 +1,49 @@
+OpenMDW License Agreement, version 1.1 (OpenMDW-1.1)
+
+By exercising rights granted to you under this agreement, you accept and agree
+to its terms.
+
+As used in this agreement, "Model Materials" means the materials provided to
+you under this agreement, consisting of: (1) one or more machine learning
+models (including architecture and parameters); and (2) all related artifacts
+(including associated data, documentation and software) that are provided to
+you hereunder.
+
+Subject to your compliance with this agreement, permission is hereby granted,
+free of charge, to deal in the Model Materials without restriction, including
+under all copyright, patent, database, and trade secret rights included or
+embodied therein.
+
+If you distribute any portion of the Model Materials, you shall retain in your
+distribution (1) a copy of this agreement, and (2) all copyright notices and
+other notices of origin included in the Model Materials that are applicable to
+your distribution.
+
+If you file, maintain, or voluntarily participate in a lawsuit against any
+person or entity asserting that the Model Materials directly or indirectly
+infringe any patent or copyright, then all rights and grants made to you
+hereunder are terminated, unless that lawsuit was in response to a
+corresponding lawsuit first brought against you.
+
+This agreement does not impose any restrictions or obligations with respect to
+any use, modification, or sharing of any outputs generated by using the Model
+Materials.
+
+THE MODEL MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
+OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE
+ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL
+TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
+
+YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY
+APPLY TO THE MODEL MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION
+ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MODEL
+MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS
+REQUIRED FOR ANY USE OF THE MODEL MATERIALS; OR (3) PERFORMING ANY DUE
+DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MODEL MATERIALS OR
+ANYTHING INCORPORATED OR EMBODIED THEREIN.
+
+IN NO EVENT SHALL THE PROVIDERS OF THE MODEL MATERIALS BE LIABLE FOR ANY CLAIM,
+DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE MODEL MATERIALS, THE
+USE THEREOF OR OTHER DEALINGS THEREIN.

PRIVACY.md

@@ -0,0 +1,6 @@
+## Privacy
+| Privacy Information |
+|---|
+| The model was trained on large-scale publicly available data that may contain images, audio-video, and text relating to people. NVIDIA collected and used this data in compliance with applicable data protection and privacy laws. This model was not designed to derive insights or otherwise learn from any personal data contained in the datasets. |
+| NVIDIA uses a combination of filters, data minimization techniques, and other guardrails to help prevent personal data from being recited by our models. We employ automated tools and data processing techniques during pre-training or training to identify and filter certain categories of personal data. For example, for text-bearing source and document components, our automated tools identified potential personal data such as person names, locations, and possible business or public-facing contact information such as email addresses and phone numbers.  We reviewed and removed any verified instances of personal data through a combination of automated filtering and human-in-the-loop validation. |
+| Please review NVIDIA's [Privacy Policy](https://www.nvidia.com/en-us/about-nvidia/privacy-policy/) for more information. |

README.md

@@ -1,3 +1,170 @@
 ---
-license: openmdw-1.1
+license: other
+license_name: openmdw-1.1
+license_link: https://openmdw.ai/license/1-1/
+base_model: nvidia/Cosmos3-Super
+library_name: diffusers
+pipeline_tag: image-to-video
+tags:
+  - nvidia
+  - cosmos3
+  - world-model
+  - omnimodel
+  - diffusion
+  - text-to-image
+  - text-to-video
+  - image-to-video
+  - quantized
+  - modelopt
+  - fp8
+  - blackwell
 ---
+
+# Cosmos3-Super — Weight-Only FP8 (NVIDIA ModelOpt)
+
+Weight-only quantization of the `Cosmos3OmniTransformer` from NVIDIA's
+[`nvidia/Cosmos3-Super`](https://huggingface.co/nvidia/Cosmos3-Super) — the 64B
+omnimodal Cosmos 3 world model (text-to-image, text-to-video, image-to-video,
+optional synchronized sound). Produced with
+[NVIDIA TensorRT Model Optimizer (ModelOpt)](https://github.com/NVIDIA/TensorRT-Model-Optimizer)
+on a single 96 GB workstation GPU, via a streaming method that never materializes
+the ~128 GB bf16 model (method scripts included).
+
+> **Only the transformer is quantized.** The VAEs and tokenizers are the original
+> bf16 components, bundled so the repo is self-contained. Loading requires the
+> bundled `load_cosmos3_modelopt.py` (see *How to use*).
+
+## Variants & measured performance
+
+Measured on an RTX 6000 Pro Blackwell (96 GB), 1024×1024 single-frame render,
+50 steps. Drop-in loading of these repos performs identically to the in-memory
+quantization path they were validated against.
+
+| Build                | Bits (weights)        | Repo size | Resident VRAM | s/it (1024² still) |
+| -------------------- | --------------------- | --------- | ------------- | ------------------ |
+| FP8 **(this repo)**  | 8-bit (E4M3)          | ~64 GB    | ~67 GB (meas.)| **~1.2**           |
+| [NVFP4 (sibling)](https://huggingface.co/prometheusAIR/Cosmos3-Super-nvfp4) | 4-bit (E2M1 + scales) | ~36 GB | ~43 GB (meas.) | ~4.6 |
+
+**Pick FP8 if it fits** — in this serving path it is both higher fidelity *and*
+~4× faster, because FP8 dequant is a single cheap scale on a native float8
+tensor, while NVFP4 dequant must unpack two 4-bit values per byte and apply
+two-level block scales in PyTorch. **Pick NVFP4 for footprint** (it brings the
+model into ~48 GB-card territory for stills). Note this is dequant-on-the-fly:
+quantization here buys **memory, not speed** — NVFP4's hardware FP4 tensor-core
+advantage only materializes in engines with FP4 GEMM kernels (TRT-LLM/vLLM
+territory), not in diffusers.
+
+Layers kept in **bf16** (not quantized): embeddings, norms, the reasoner head,
+in/out projections, time/modality adapters, audio adapter. The 64 transformer
+blocks' attention + MLP linears (incl. MoE experts) are quantized.
+
+## Status
+
+- ✅ **Drop-in loading verified** end to end (load → render → performance parity
+  with the in-memory method) on Blackwell (sm_120), **via the bundled loader**.
+- ✅ `modelopt_state.pth` is part of the checkpoint and is **required** — it
+  restores the quantized module structure at load. Do not delete it.
+- ⚠️ The loader (`load_cosmos3_modelopt.py`) is **required**, not optional. The
+  current diffusers/accelerate/modelopt combination cannot materialize a
+  pre-quantized ModelOpt checkpoint unaided; the loader applies three small,
+  source-verified workarounds (parameter materialization for packed weights,
+  payload-dtype restoration for FP8, and weight-only quantizer enforcement)
+  plus the validated bf16 dtype normalization. ModelOpt marks this path
+  experimental; expect the loader to become unnecessary as upstream catches up.
+- ❌ **vLLM-Omni:** not a working path as of 0.22.0. This is an upstream roadmap
+  gap, not a defect of this checkpoint: vLLM-Omni's ModelOpt integration is
+  currently wired for LLMs only, and ModelOpt-quantized diffusion support is an
+  open RFC ([#2709](https://github.com/vllm-project/vllm-omni/issues/2709),
+  [#1959](https://github.com/vllm-project/vllm-omni/issues/1959)).
+- ❌ **ComfyUI:** no known node support for this ModelOpt layout (the NF4 build
+  linked below has community nodes; this one does not).
+- Validated only on Blackwell. FP8 on Hopper/Ada is plausible but unverified
+  here.
+
+## How to use
+
+Requires a `diffusers` build with Cosmos 3 support (currently from source) plus
+`modelopt` and `accelerate`. Pin to the verified versions for guaranteed
+reproducibility (newer versions may also work, but this code path moves fast):
+
+```bash
+pip install "git+https://github.com/huggingface/diffusers.git@2c7efb95349296cf6bcce981ea036275a82a94df"
+pip install accelerate "nvidia-modelopt==0.44.0"
+```
+
+```python
+from load_cosmos3_modelopt import load_pipe   # bundled in this repo
+from diffusers import UniPCMultistepScheduler
+
+pipe = load_pipe("prometheusAIR/Cosmos3-Super-fp8")   # or a local path
+pipe.scheduler = UniPCMultistepScheduler.from_config(
+    pipe.scheduler.config, flow_shift=3.0   # NVIDIA's text-to-image setting; use 5.0 for image-to-video
+)
+
+# Single image -- pass parameters EXPLICITLY (see warning below):
+r = pipe("a weathered lighthouse on a cliff at golden hour, photoreal, 50mm",
+         height=1024, width=1024, num_frames=1,
+         num_inference_steps=50, guidance_scale=4.0)
+r.video[0].save("out.png")   # .video is the list of PIL frames; [0] is the image
+
+# Video (~2 s): frame counts of the form 4n+1 map cleanly to the VAE's 4x
+# temporal compression; 24 fps is the native rate and conditions the model.
+r = pipe("The lighthouse beam sweeps slowly across the water. Static camera.",
+         height=704, width=1280, num_frames=49, fps=24.0,
+         num_inference_steps=35, guidance_scale=6.0)
+```
+
+These still-image settings (1024², 50 steps, guidance 4.0, `flow_shift=3.0`,
+`result.video[0]`) match NVIDIA's first-party Cosmos3 text-to-image reference.
+
+> ⚠️ **A bare `pipe(prompt)` call renders a 189-frame 720×1280 video** (~8 s at
+> 24 fps) — that is the pipeline's built-in default, not a still. It takes ~40×
+> the compute of a single frame and is the most common reason this model
+> "seems slow." Always pass `num_frames`/`height`/`width` explicitly.
+
+Cosmos 3 expects a dense structured-JSON prompt for best quality; plain prompts
+work but render softer. See NVIDIA's prompt-upsampling docs.
+
+**Reproducing from scratch:** `quantize_cosmos3_super_streaming.py` (included)
+streams the bf16 shards directly into compressed FP8/NVFP4 form (peak memory ≈
+the compressed footprint, so a single 96 GB card suffices), and
+`repackage_for_hf.py` emits this repo's round-trippable layout via
+`save_pretrained` + `enable_huggingface_checkpointing()` — note that ModelOpt's
+`export_hf_checkpoint()` produces a *deployment* checkpoint that diffusers
+cannot round-trip; the `modelopt_state.pth` from `save_pretrained` is what makes
+drop-in loading possible. `serve_cosmos3_diffusers.py` is a small FastAPI server
+(text→image, image→video) around the same model.
+
+## Known limitations / caveats
+
+- **The bundled loader is required** (see *Status*). FP8 additionally depends on
+  its payload-dtype restoration: diffusers' loader casts floating params to
+  `torch_dtype` when no hf_quantizer is present (flagged by a TODO in diffusers'
+  own source), which would otherwise corrupt float8 payloads.
+- **QKV scale unification was skipped at export** (ModelOpt's fusion probe
+  doesn't recognize this architecture); q/k/v keep independent scales. Harmless
+  here; relevant only to engines that fuse QKV.
+- Render sharpness depends heavily on prompt density, scheduler settings, and
+  guidance — tune these; they are not quantization loss.
+
+## Guardrails
+
+Cosmos 3 ships an optional safety checker (`cosmos_guardrail`). The bundled
+loader passes `enable_safety_checker=False` for local single-user use. If you
+deploy this or publish generated media, install `cosmos-guardrail`, accept the
+gated [`nvidia/Cosmos-Guardrail1`](https://huggingface.co/nvidia/Cosmos-Guardrail1)
+model (released under its own NVIDIA Open Model License, separate from this
+repo's OpenMDW-1.1), and run with `load_pipe(..., enable_safety_checker=True)`.
+
+## Provenance & License
+
+- **Derivative of:** [`nvidia/Cosmos3-Super`](https://huggingface.co/nvidia/Cosmos3-Super) (bf16). This repo modifies only the weight encoding of the transformer.
+- **Produced with:** NVIDIA TensorRT Model Optimizer + diffusers (from source).
+- **Exact versions used:** `diffusers 0.39.0.dev0` @ [`2c7efb9`](https://github.com/huggingface/diffusers/commit/2c7efb95349296cf6bcce981ea036275a82a94df), `nvidia-modelopt 0.44.0`, `accelerate 1.13.0`, `torch 2.12.0`, CUDA 13.3.
+- **License:** [OpenMDW-1.1](https://openmdw.ai/license/1-1/), inherited from the base model. This repo includes a copy of the agreement (`LICENSE`) and documents its origin above; the upstream repo ships no separate NOTICE file. OpenMDW-1.1 permits modification and redistribution and places no restrictions on generated outputs; you remain responsible for clearing any third-party rights embodied in the materials.
+
+## Related repos
+
+- **Sibling NVFP4 build (smaller footprint, ~36 GB):** [`prometheusAIR/Cosmos3-Super-nvfp4`](https://huggingface.co/prometheusAIR/Cosmos3-Super-nvfp4)
+- **Original (bf16, source):** [`nvidia/Cosmos3-Super`](https://huggingface.co/nvidia/Cosmos3-Super)
+- **NF4 (bitsandbytes; broad GPU compatibility incl. Ampere/Ada; drop-in + ComfyUI nodes):** [`SanDiegoDude/Cosmos3-Super-nf4`](https://huggingface.co/SanDiegoDude/Cosmos3-Super-nf4) — a good choice if you are not on Blackwell-class hardware or want turnkey ComfyUI support.

SAFETY.md

@@ -0,0 +1,11 @@
+## Safety & Security
+
+| Field | Response |
+| :---- | :---- |
+| Model Application(s) | World reasoning and generation for Physical AI. |
+| Describe the life critical impact: | This model is not a safety-certified component and must not be used as the sole basis for life-critical decisions or control without additional system-level validation, safety analysis, and safeguards. The model is not designed or tested by NVIDIA for use in any system or application where the use of or failure of such system or application developed with the model could result in injury, death, or catastrophic damage. NVIDIA is not liable to any party, in whole or in part, for any claims or damages arising from those uses. Any system or application developed with the model must include sufficient safety and redundancy features and comply with applicable legal and regulatory standards and requirements. |
+| Description of methods implemented in data acquisition or processing, if any, to address other types of potentially harmful data in the training, testing, and validation data:  | Training, evaluation, and validation datasets pass through multi-stage automated and manual filtering to reduce harmful, unsafe, restricted, or policy-violating content. Pipelines include source-licensing review, deduplication, metadata-based and classifier-based moderation, embedding-based anomaly detection, and human audits on selected datasets. For Physical AI data (robotics, autonomous driving, industrial scenes), filtering also targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark and red-team testing surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No data-filtering process can guarantee complete removal; developers are responsible for application-specific safeguards and validation before deployment. |
+| Description of any methods implemented in data acquisition or processing, if any, to address illegal or harmful content in the training data, including, but not limited to, child sexual abuse material (CSAM) and non-consensual intimate imagery (NCII)  | In addition to the general unsafe-content filtering described above, training data acquisition and preprocessing apply CSAM- and NCII-specific safeguards: hash-matching systems against known CSAM databases, classifier-based moderation models trained specifically for explicit content and NCII detection, and provenance and licensing review for sources containing human imagery. Identified content is removed at ingest, with human review and targeted audits supplementing automated filtering for selected datasets. Despite these safeguards, no large-scale data-filtering system can guarantee complete detection. Ongoing monitoring and dataset review continue post-release. |
+| Use Case Restrictions | Use is governed by the [OpenMDW1.1](https://openmdw.ai/) |
+| Model and dataset restrictions | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to. |
+| Responsible Data Handling | This AI model was developed based on our policies to ensure responsible data handling and risk mitigation. The datasets used for training have been scanned for harmful content and illegal content, consistent with our policies including scanning for Child Sexual Abuse Material (CSAM). Ongoing review and monitoring mechanisms are in place based on our policies and to maintain data integrity. |

assets/example_action_fd_agibotworld_4chunk_output.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b01266b6cd27478514133b00ada9c33db3f9444167f09942c11f880c629c8c0
+size 672918

assets/example_action_fd_agibotworld_action_chunks.json

@@ -0,0 +1,2008 @@
+{
+  "prompt": "Pickup items in the supermarket",
+  "domain_name": "agibotworld",
+  "view_point": "concat_view",
+  "fps": 10,
+  "image_size": 480,
+  "action_chunk_size": 16,
+  "num_chunks": 4,
+  "action_shape_per_chunk": [
+    16,
+    29
+  ],
+  "description": "Four normalized 29-D AgiBotWorld-Beta action chunks. Use the first frame as the initial conditioning image. For chunks 1-3, condition on the final generated frame from the previous chunk.",
+  "action_chunks": [
+    [
+      [
+        0.009043693542480469,
+        0.20256006717681885,
+        0.04290425777435303,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.010607659816741943,
+        0.005519270896911621,
+        0.04532516002655029,
+        1.0,
+        2.4557113647460938e-05,
+        2.968311309814453e-05,
+        -2.4557113647460938e-05,
+        1.0,
+        2.181529998779297e-05,
+        0.41925930976867676,
+        0.033257126808166504,
+        0.0046384334564208984,
+        0.03365051746368408,
+        1.0,
+        -8.106231689453125e-06,
+        7.033348083496094e-06,
+        8.106231689453125e-06,
+        1.0,
+        4.792213439941406e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009057760238647461,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.010164499282836914,
+        0.006109714508056641,
+        0.04489457607269287,
+        1.0,
+        -2.1159648895263672e-05,
+        -1.5616416931152344e-05,
+        2.110004425048828e-05,
+        1.0,
+        -1.7881393432617188e-06,
+        0.41925930976867676,
+        0.033304810523986816,
+        0.005009293556213379,
+        0.03424406051635742,
+        1.0,
+        1.7881393432617188e-06,
+        -3.2186508178710938e-06,
+        -1.8477439880371094e-06,
+        1.0,
+        2.4437904357910156e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009057760238647461,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.01051628589630127,
+        0.006176114082336426,
+        0.04440665245056152,
+        1.0,
+        3.147125244140625e-05,
+        3.0994415283203125e-06,
+        -3.147125244140625e-05,
+        1.0,
+        -1.6987323760986328e-05,
+        0.41925930976867676,
+        0.03370165824890137,
+        0.0064890384674072266,
+        0.03395986557006836,
+        1.0,
+        -3.349781036376953e-05,
+        -1.424551010131836e-05,
+        3.349781036376953e-05,
+        1.0,
+        -6.020069122314453e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009057760238647461,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.010625958442687988,
+        0.006899237632751465,
+        0.044704556465148926,
+        1.0,
+        3.254413604736328e-05,
+        -1.043081283569336e-05,
+        -3.254413604736328e-05,
+        1.0,
+        -1.3887882232666016e-05,
+        0.41925930976867676,
+        0.03398740291595459,
+        0.0045958757400512695,
+        0.03477442264556885,
+        1.0,
+        2.9325485229492188e-05,
+        -3.647804260253906e-05,
+        -2.9325485229492188e-05,
+        1.0,
+        2.3365020751953125e-05,
+        0.41629624366760254
+      ],
+      [
+        0.01160132884979248,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        9.5367431640625e-07,
+        -2.5033950805664062e-06,
+        -9.5367431640625e-07,
+        1.0,
+        0.0,
+        -0.009872138500213623,
+        0.005211949348449707,
+        0.045044898986816406,
+        1.0,
+        -3.808736801147461e-05,
+        -3.814697265625e-06,
+        3.814697265625e-05,
+        1.0,
+        -5.4836273193359375e-06,
+        0.41925930976867676,
+        0.03165924549102783,
+        0.0048122406005859375,
+        0.0322871208190918,
+        1.0,
+        -1.7881393432617188e-06,
+        7.87973403930664e-05,
+        1.7881393432617188e-06,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.01051628589630127,
+        0.0053795576095581055,
+        0.045049309730529785,
+        1.0,
+        -4.76837158203125e-06,
+        9.5367431640625e-06,
+        4.76837158203125e-06,
+        1.0,
+        3.504753112792969e-05,
+        0.41925930976867676,
+        0.034897446632385254,
+        0.005955934524536133,
+        0.0352022647857666,
+        1.0,
+        1.1920928955078125e-05,
+        -6.92605972290039e-05,
+        -1.1920928955078125e-05,
+        1.0,
+        -2.1636486053466797e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009052157402038574,
+        0.20257973670959473,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.010749280452728271,
+        0.007395386695861816,
+        0.0448070764541626,
+        1.0,
+        2.384185791015625e-06,
+        1.4185905456542969e-05,
+        -2.384185791015625e-06,
+        1.0,
+        -4.285573959350586e-05,
+        0.41925930976867676,
+        0.031437039375305176,
+        0.0054999589920043945,
+        0.03345012664794922,
+        1.0,
+        -2.47955322265625e-05,
+        8.356571197509766e-05,
+        2.47955322265625e-05,
+        1.0,
+        -1.710653305053711e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20257973670959473,
+        0.04291260242462158,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.009954392910003662,
+        0.005501866340637207,
+        0.04485750198364258,
+        1.0,
+        -2.002716064453125e-05,
+        -2.4437904357910156e-05,
+        2.002716064453125e-05,
+        1.0,
+        2.8252601623535156e-05,
+        0.41925930976867676,
+        0.03527843952178955,
+        0.005295157432556152,
+        0.03457236289978027,
+        1.0,
+        -1.049041748046875e-05,
+        -8.213520050048828e-05,
+        1.049041748046875e-05,
+        1.0,
+        3.4332275390625e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009052157402038574,
+        0.20257973670959473,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.0008448958396911621,
+        -0.017412126064300537,
+        0.05560195446014404,
+        0.9999996423721313,
+        -0.0009481906890869141,
+        -5.638599395751953e-05,
+        0.0009481906890869141,
+        0.9999994039535522,
+        0.0004875659942626953,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005210161209106445,
+        0.0339665412902832,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.03963160514831543,
+        -0.046320974826812744,
+        0.07942080497741699,
+        0.9999939203262329,
+        -0.0032505393028259277,
+        -0.0012335777282714844,
+        0.0032502412796020508,
+        0.9999947547912598,
+        -0.0002353191375732422,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005217909812927246,
+        0.03396797180175781,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.00911402702331543,
+        0.20263886451721191,
+        0.04292500019073486,
+        1.0,
+        0.0,
+        -5.960464477539063e-08,
+        0.0,
+        1.0,
+        -2.682209014892578e-06,
+        0.09385919570922852,
+        -0.05113154649734497,
+        0.09816932678222656,
+        0.9999802112579346,
+        -0.005321979522705078,
+        -0.0033246874809265137,
+        0.005314230918884277,
+        0.9999830722808838,
+        -0.0023511648178100586,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005229473114013672,
+        0.033969879150390625,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.009057760238647461,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.12021923065185547,
+        -0.03888678550720215,
+        0.11576962471008301,
+        0.9999756813049316,
+        -0.005499899387359619,
+        -0.004302918910980225,
+        0.005483388900756836,
+        0.9999775886535645,
+        -0.003832995891571045,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005214095115661621,
+        0.03396737575531006,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.009049415588378906,
+        0.20256006717681885,
+        0.04290628433227539,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.1539161205291748,
+        -0.02273625135421753,
+        0.12730515003204346,
+        0.9999697208404541,
+        -0.005183577537536621,
+        -0.00581127405166626,
+        0.005152106285095215,
+        0.9999721050262451,
+        -0.005426645278930664,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005210161209106445,
+        0.0339665412902832,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.011651992797851562,
+        0.2025994062423706,
+        0.04295003414154053,
+        1.0,
+        9.5367431640625e-07,
+        -2.5033950805664062e-06,
+        -9.5367431640625e-07,
+        1.0,
+        -9.5367431640625e-07,
+        0.1874760389328003,
+        -0.02913987636566162,
+        0.13934898376464844,
+        0.999955415725708,
+        -0.0063391923904418945,
+        -0.007002890110015869,
+        0.0062923431396484375,
+        0.9999579191207886,
+        -0.0066814422607421875,
+        0.41925930976867676,
+        0.03327834606170654,
+        0.005256533622741699,
+        0.0339813232421875,
+        1.0,
+        -7.748603820800781e-07,
+        -5.960464477539063e-08,
+        8.344650268554688e-07,
+        1.0,
+        -5.364418029785156e-07,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.20743560791015625,
+        -0.020664572715759277,
+        0.17266845703125,
+        0.9999444484710693,
+        -0.007667183876037598,
+        -0.00723874568939209,
+        0.007605195045471191,
+        0.9999344348907471,
+        -0.008556544780731201,
+        0.41925930976867676,
+        0.03328895568847656,
+        0.005217909812927246,
+        0.03396928310394287,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.31465721130371094,
+        -0.045416176319122314,
+        0.24895763397216797,
+        0.9998956918716431,
+        -0.010226964950561523,
+        -0.010205447673797607,
+        0.010100603103637695,
+        0.9998726844787598,
+        -0.012354671955108643,
+        0.41925930976867676,
+        0.03328895568847656,
+        0.005217909812927246,
+        0.03396928310394287,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ]
+    ],
+    [
+      [
+        0.009052157402038574,
+        0.20257973670959473,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.010411202907562256,
+        0.006074786186218262,
+        0.044891953468322754,
+        1.0,
+        0.00010097026824951172,
+        1.5497207641601562e-06,
+        -0.00010102987289428711,
+        1.0,
+        1.1920928955078125e-06,
+        0.41925930976867676,
+        0.034119606018066406,
+        0.0051136016845703125,
+        0.034647583961486816,
+        1.0,
+        3.62396240234375e-05,
+        -4.118680953979492e-05,
+        -3.618001937866211e-05,
+        1.0,
+        -1.3947486877441406e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20257973670959473,
+        0.04291260242462158,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.010630488395690918,
+        0.006099224090576172,
+        0.04487788677215576,
+        1.0,
+        -5.799531936645508e-05,
+        2.002716064453125e-05,
+        5.793571472167969e-05,
+        1.0,
+        -1.3232231140136719e-05,
+        0.41925930976867676,
+        0.03215658664703369,
+        0.00477755069732666,
+        0.032892584800720215,
+        1.0,
+        2.384185791015625e-06,
+        5.4001808166503906e-05,
+        -2.384185791015625e-06,
+        1.0,
+        1.2516975402832031e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009066224098205566,
+        0.2025994062423706,
+        0.042914628982543945,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        -0.009620904922485352,
+        0.0038878917694091797,
+        0.046347856521606445,
+        1.0,
+        -0.00011581182479858398,
+        -1.0132789611816406e-06,
+        0.00011587142944335938,
+        1.0,
+        7.665157318115234e-05,
+        0.41925930976867676,
+        0.034696340560913086,
+        0.006465911865234375,
+        0.03510606288909912,
+        1.0,
+        -3.439188003540039e-05,
+        -5.424022674560547e-05,
+        3.445148468017578e-05,
+        1.0,
+        -5.364418029785156e-06,
+        0.41629624366760254
+      ],
+      [
+        0.009055018424987793,
+        0.20256006717681885,
+        0.04290628433227539,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.009932160377502441,
+        -0.03261244297027588,
+        0.06218290328979492,
+        0.9999983310699463,
+        -0.0017368793487548828,
+        -0.0003128647804260254,
+        0.0017371177673339844,
+        0.9999983310699463,
+        0.0006134510040283203,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005214095115661621,
+        0.03396737575531006,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.05640697479248047,
+        -0.046244144439697266,
+        0.08853423595428467,
+        0.999990701675415,
+        -0.003899216651916504,
+        -0.001820981502532959,
+        0.0038973093032836914,
+        0.9999918937683105,
+        -0.0010201334953308105,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005217909812927246,
+        0.03396797180175781,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.009085893630981445,
+        0.20261919498443604,
+        0.042914628982543945,
+        1.0,
+        0.0,
+        -5.960464477539063e-08,
+        0.0,
+        1.0,
+        -2.682209014892578e-06,
+        0.10252106189727783,
+        -0.050013601779937744,
+        0.10689389705657959,
+        0.9999780654907227,
+        -0.00561290979385376,
+        -0.003548860549926758,
+        0.005602836608886719,
+        0.9999803304672241,
+        -0.0028455257415771484,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005210161209106445,
+        0.0339665412902832,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.00907754898071289,
+        0.2025994062423706,
+        0.04291880130767822,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.13923311233520508,
+        -0.03134775161743164,
+        0.11226391792297363,
+        0.9999709129333496,
+        -0.005446195602416992,
+        -0.00535351037979126,
+        0.005421996116638184,
+        0.9999749660491943,
+        -0.004530847072601318,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005225658416748047,
+        0.03396928310394287,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.011612653732299805,
+        0.20257973670959473,
+        0.04290628433227539,
+        1.0,
+        9.5367431640625e-07,
+        -2.5033950805664062e-06,
+        -9.5367431640625e-07,
+        1.0,
+        0.0,
+        0.1607367992401123,
+        -0.0290105938911438,
+        0.1363682746887207,
+        0.9999675750732422,
+        -0.00552743673324585,
+        -0.0058705806732177734,
+        0.005494356155395508,
+        0.9999691247940063,
+        -0.005625605583190918,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005217909812927246,
+        0.03396797180175781,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.009085893630981445,
+        0.20257973670959473,
+        0.042948007583618164,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -1.0132789611816406e-06,
+        0.1929306983947754,
+        -0.014666199684143066,
+        0.15205836296081543,
+        0.9999532699584961,
+        -0.006599485874176025,
+        -0.007064223289489746,
+        0.006544232368469238,
+        0.9999480247497559,
+        -0.007818400859832764,
+        0.41925930976867676,
+        0.03327834606170654,
+        0.005248904228210449,
+        0.03397989273071289,
+        1.0,
+        -7.748603820800781e-07,
+        -5.960464477539063e-08,
+        8.344650268554688e-07,
+        1.0,
+        -5.364418029785156e-07,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.30631983280181885,
+        -0.04655855894088745,
+        0.2282034158706665,
+        0.9998886585235596,
+        -0.01070857048034668,
+        -0.010384440422058105,
+        0.010587453842163086,
+        0.9998760223388672,
+        -0.011655926704406738,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005214095115661621,
+        0.03396880626678467,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.3243972063064575,
+        -0.058551788330078125,
+        0.2604410648345947,
+        0.9998958110809326,
+        -0.010265886783599854,
+        -0.010132789611816406,
+        0.010138750076293945,
+        0.9998701810836792,
+        -0.01252216100692749,
+        0.41925930976867676,
+        0.03328359127044678,
+        0.00520634651184082,
+        0.033963799476623535,
+        1.0,
+        4.76837158203125e-07,
+        3.5762786865234375e-07,
+        -4.172325134277344e-07,
+        1.0,
+        3.5762786865234375e-07,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.331770658493042,
+        -0.0866466760635376,
+        0.27182281017303467,
+        0.9998891353607178,
+        -0.011347353458404541,
+        -0.009640693664550781,
+        0.011227130889892578,
+        0.9998596906661987,
+        -0.012438654899597168,
+        0.41925930976867676,
+        0.033225417137145996,
+        0.005071163177490234,
+        0.033928513526916504,
+        1.0,
+        5.0067901611328125e-06,
+        2.7418136596679688e-06,
+        -5.0067901611328125e-06,
+        1.0,
+        4.0531158447265625e-06,
+        0.41629624366760254
+      ],
+      [
+        0.009066224098205566,
+        0.2025994062423706,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.36654579639434814,
+        -0.0934380292892456,
+        0.30077290534973145,
+        0.9997755289077759,
+        -0.018354415893554688,
+        -0.010581493377685547,
+        0.01818561553955078,
+        0.9997092485427856,
+        -0.01583230495452881,
+        0.41925930976867676,
+        0.03137350082397461,
+        0.0008289813995361328,
+        0.03272604942321777,
+        1.0,
+        0.0001519918441772461,
+        8.416175842285156e-05,
+        -0.00015205144882202148,
+        1.0,
+        0.00012350082397460938,
+        0.41629624366760254
+      ],
+      [
+        0.009055018424987793,
+        0.20257973670959473,
+        0.04290628433227539,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.3391350507736206,
+        -0.11247771978378296,
+        0.30669987201690674,
+        0.9997462034225464,
+        -0.02040243148803711,
+        -0.009555697441101074,
+        0.020258665084838867,
+        0.9996836185455322,
+        -0.014911115169525146,
+        0.41925930976867676,
+        0.03739488124847412,
+        0.014652729034423828,
+        0.03664207458496094,
+        1.0,
+        -0.00032657384872436523,
+        -0.00018095970153808594,
+        0.00032651424407958984,
+        0.9999998807907104,
+        -0.00026535987854003906,
+        0.41629624366760254
+      ],
+      [
+        0.009260416030883789,
+        0.2028360366821289,
+        0.04296672344207764,
+        1.0,
+        1.1920928955078125e-07,
+        -2.980232238769531e-07,
+        -1.1920928955078125e-07,
+        1.0,
+        -1.5914440155029297e-05,
+        0.33494579792022705,
+        -0.07520890235900879,
+        0.2943739891052246,
+        0.9997798204421997,
+        -0.01838594675064087,
+        -0.010106980800628662,
+        0.018214821815490723,
+        0.9996933937072754,
+        -0.016768813133239746,
+        0.41925930976867676,
+        0.024235844612121582,
+        -0.014741063117980957,
+        0.02833724021911621,
+        0.9999996423721313,
+        0.0006879568099975586,
+        0.0004069805145263672,
+        -0.0006881952285766602,
+        0.9999995231628418,
+        0.0005651712417602539,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20256006717681885,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.29727888107299805,
+        0.02291703224182129,
+        0.24155020713806152,
+        0.999849796295166,
+        -0.014276325702667236,
+        -0.00983363389968872,
+        0.014076709747314453,
+        0.9996992349624634,
+        -0.020082950592041016,
+        0.41925930976867676,
+        0.040045738220214844,
+        0.019493699073791504,
+        0.038599610328674316,
+        0.9999997615814209,
+        -0.00048214197158813477,
+        -0.0002892613410949707,
+        0.0004819631576538086,
+        0.9999997615814209,
+        -0.00040209293365478516,
+        0.41629624366760254
+      ]
+    ],
+    [
+      [
+        0.009085893630981445,
+        0.20261919498443604,
+        0.042914628982543945,
+        1.0,
+        0.0,
+        -5.960464477539063e-08,
+        0.0,
+        1.0,
+        -2.682209014892578e-06,
+        0.11258077621459961,
+        -0.04367637634277344,
+        0.10712110996246338,
+        0.999976396560669,
+        -0.005554258823394775,
+        -0.004074573516845703,
+        0.005540609359741211,
+        0.9999790191650391,
+        -0.003357231616973877,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005210161209106445,
+        0.0339665412902832,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.00907754898071289,
+        0.2025994062423706,
+        0.04291880130767822,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.14761626720428467,
+        -0.03002721071243286,
+        0.12956809997558594,
+        0.9999697208404541,
+        -0.005600392818450928,
+        -0.005403280258178711,
+        0.0055730342864990234,
+        0.9999716281890869,
+        -0.005077004432678223,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005225658416748047,
+        0.03396928310394287,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.011612653732299805,
+        0.20257973670959473,
+        0.04290628433227539,
+        1.0,
+        9.5367431640625e-07,
+        -2.5033950805664062e-06,
+        -9.5367431640625e-07,
+        1.0,
+        0.0,
+        0.17086505889892578,
+        -0.018072426319122314,
+        0.13277769088745117,
+        0.999964714050293,
+        -0.005352973937988281,
+        -0.00646662712097168,
+        0.005311727523803711,
+        0.9999654293060303,
+        -0.006388425827026367,
+        0.41925930976867676,
+        0.0332942008972168,
+        0.005217909812927246,
+        0.03396797180175781,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.41629624366760254
+      ],
+      [
+        0.009085893630981445,
+        0.20257973670959473,
+        0.042948007583618164,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -1.0132789611816406e-06,
+        0.20010781288146973,
+        -0.0246087908744812,
+        0.15921831130981445,
+        0.9999470710754395,
+        -0.00736159086227417,
+        -0.007179737091064453,
+        0.0073053836822509766,
+        0.9999427795410156,
+        -0.00782400369644165,
+        0.41925930976867676,
+        0.03327834606170654,
+        0.005248904228210449,
+        0.03397989273071289,
+        1.0,
+        -7.748603820800781e-07,
+        -5.960464477539063e-08,
+        8.344650268554688e-07,
+        1.0,
+        -5.364418029785156e-07,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.31356537342071533,
+        -0.04342836141586304,
+        0.2481241226196289,
+        0.9998904466629028,
+        -0.010683715343475342,
+        -0.01024484634399414,
+        0.01055598258972168,
+        0.9998668432235718,
+        -0.01244121789932251,
+        0.41925930976867676,
+        0.03328895568847656,
+        0.005221843719482422,
+        0.03397011756896973,
+        1.0,
+        -1.7881393432617188e-07,
+        0.0,
+        1.1920928955078125e-07,
+        1.0,
+        -1.1920928955078125e-07,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.33043670654296875,
+        -0.07546043395996094,
+        0.26550519466400146,
+        0.9998939037322998,
+        -0.010528624057769775,
+        -0.010073602199554443,
+        0.010405302047729492,
+        0.9998712539672852,
+        -0.012217998504638672,
+        0.41925930976867676,
+        0.03329956531524658,
+        0.005252718925476074,
+        0.03398323059082031,
+        1.0,
+        -1.1920928955078125e-06,
+        -7.748603820800781e-07,
+        1.1920928955078125e-06,
+        1.0,
+        -1.0132789611816406e-06,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.3373715877532959,
+        -0.09454900026321411,
+        0.26748788356781006,
+        0.9998712539672852,
+        -0.012611329555511475,
+        -0.009919703006744385,
+        0.01248788833618164,
+        0.999845027923584,
+        -0.012407541275024414,
+        0.41925930976867676,
+        0.033394813537597656,
+        0.005472898483276367,
+        0.0340428352355957,
+        1.0,
+        -8.881092071533203e-06,
+        -4.887580871582031e-06,
+        8.940696716308594e-06,
+        1.0,
+        -7.212162017822266e-06,
+        0.41629624366760254
+      ],
+      [
+        0.009066224098205566,
+        0.2025994062423706,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.3717309236526489,
+        -0.0959848165512085,
+        0.3111867904663086,
+        0.9997556209564209,
+        -0.019321084022521973,
+        -0.010733485221862793,
+        0.019145727157592773,
+        0.9996854066848755,
+        -0.016201794147491455,
+        0.41925930976867676,
+        0.0340878963470459,
+        0.007068514823913574,
+        0.03449106216430664,
+        1.0,
+        -6.383657455444336e-05,
+        -3.534555435180664e-05,
+        6.389617919921875e-05,
+        1.0,
+        -5.1856040954589844e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009055018424987793,
+        0.20257973670959473,
+        0.04290628433227539,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.3436669111251831,
+        -0.10583305358886719,
+        0.3016718626022339,
+        0.9997358322143555,
+        -0.020739614963531494,
+        -0.009907543659210205,
+        0.020581960678100586,
+        0.9996639490127563,
+        -0.01575833559036255,
+        0.41925930976867676,
+        0.030542850494384766,
+        -0.0011028051376342773,
+        0.03217768669128418,
+        1.0,
+        0.0002186298370361328,
+        0.00012099742889404297,
+        -0.0002186298370361328,
+        1.0,
+        0.00017762184143066406,
+        0.41629624366760254
+      ],
+      [
+        0.009260416030883789,
+        0.2028360366821289,
+        0.04296672344207764,
+        1.0,
+        1.1920928955078125e-07,
+        -2.980232238769531e-07,
+        -1.1920928955078125e-07,
+        1.0,
+        -1.5914440155029297e-05,
+        0.3160369396209717,
+        -0.03754878044128418,
+        0.2892639636993408,
+        0.999807596206665,
+        -0.017129719257354736,
+        -0.009551525115966797,
+        0.016954421997070312,
+        0.9996918439865112,
+        -0.018134236335754395,
+        0.41925930976867676,
+        0.03539478778839111,
+        0.009861946105957031,
+        0.03578758239746094,
+        1.0,
+        -0.00015670061111450195,
+        -8.600950241088867e-05,
+        0.00015664100646972656,
+        1.0,
+        -0.00012248754501342773,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20256006717681885,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.28902363777160645,
+        0.0013829469680786133,
+        0.23963022232055664,
+        0.9998641014099121,
+        -0.013593852519989014,
+        -0.009318351745605469,
+        0.013410806655883789,
+        0.9997210502624512,
+        -0.01943725347518921,
+        0.41925930976867676,
+        0.04227328300476074,
+        0.02469015121459961,
+        0.03992271423339844,
+        0.9999997615814209,
+        -0.0006698369979858398,
+        -0.0003890395164489746,
+        0.0006695985794067383,
+        0.9999997615814209,
+        -0.0005470514297485352,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20256006717681885,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.32505500316619873,
+        0.011157870292663574,
+        0.26268625259399414,
+        0.9998914003372192,
+        -0.00958329439163208,
+        -0.011197924613952637,
+        0.009337067604064941,
+        0.9997179508209229,
+        -0.021832704544067383,
+        0.41925930976867676,
+        0.02175426483154297,
+        -0.027143001556396484,
+        0.02682507038116455,
+        0.9999990463256836,
+        0.0012558698654174805,
+        0.0006737709045410156,
+        -0.0012564659118652344,
+        0.9999986886978149,
+        0.0009870529174804688,
+        0.41629624366760254
+      ],
+      [
+        0.009091615676879883,
+        0.20256006717681885,
+        0.04294168949127197,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -9.5367431640625e-07,
+        0.2935601472854614,
+        0.02565944194793701,
+        0.24912917613983154,
+        0.9998788833618164,
+        -0.010689914226531982,
+        -0.011303961277008057,
+        0.010461926460266113,
+        0.9997446537017822,
+        -0.020034193992614746,
+        0.41925930976867676,
+        0.0381091833114624,
+        0.015552878379821777,
+        0.03541207313537598,
+        0.9999998807907104,
+        -0.00038051605224609375,
+        -0.00010246038436889648,
+        0.00038051605224609375,
+        0.9999998807907104,
+        -0.0002970099449157715,
+        0.41629624366760254
+      ],
+      [
+        0.009063482284545898,
+        0.20257973670959473,
+        0.04291045665740967,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.28693127632141113,
+        0.1092977523803711,
+        0.31335508823394775,
+        0.9998719692230225,
+        -0.006708920001983643,
+        -0.014529645442962646,
+        0.006423592567443848,
+        0.9997873306274414,
+        -0.019595563411712646,
+        0.41925930976867676,
+        0.043781280517578125,
+        0.03405153751373291,
+        0.0410383939743042,
+        0.9999994039535522,
+        -0.0010578632354736328,
+        -0.0003858208656311035,
+        0.0010575056076049805,
+        0.9999990463256836,
+        -0.0008174777030944824,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20257973670959473,
+        0.042914628982543945,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.2642625570297241,
+        0.19131505489349365,
+        0.3423175811767578,
+        0.9998579025268555,
+        -0.005951404571533203,
+        -0.01576918363571167,
+        0.005600333213806152,
+        0.9997375011444092,
+        -0.022212564945220947,
+        0.41925930976867676,
+        0.04341089725494385,
+        0.013211607933044434,
+        0.03693842887878418,
+        1.0,
+        -0.00010991096496582031,
+        -0.00019061565399169922,
+        0.00010991096496582031,
+        1.0,
+        0.0001575946807861328,
+        0.41629624366760254
+      ],
+      [
+        0.009063482284545898,
+        0.20256006717681885,
+        0.04291260242462158,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.16873621940612793,
+        0.25091099739074707,
+        0.359053373336792,
+        0.9999058246612549,
+        -0.006036102771759033,
+        -0.012328088283538818,
+        0.005730032920837402,
+        0.999678373336792,
+        -0.02470952272415161,
+        0.41925930976867676,
+        0.022193431854248047,
+        -0.010541379451751709,
+        0.02903127670288086,
+        0.9999994039535522,
+        0.0006458759307861328,
+        0.0008552074432373047,
+        -0.0006463527679443359,
+        0.9999996423721313,
+        0.0005235671997070312,
+        0.41629624366760254
+      ]
+    ],
+    [
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.33918070793151855,
+        -0.08039677143096924,
+        0.269872784614563,
+        0.9998908042907715,
+        -0.01066964864730835,
+        -0.010227024555206299,
+        0.010541915893554688,
+        0.9998667240142822,
+        -0.012470245361328125,
+        0.41925930976867676,
+        0.03326773643493652,
+        0.005175471305847168,
+        0.03395795822143555,
+        1.0,
+        1.430511474609375e-06,
+        8.344650268554688e-07,
+        -1.430511474609375e-06,
+        1.0,
+        1.0728836059570312e-06,
+        0.41629624366760254
+      ],
+      [
+        0.00906062126159668,
+        0.2025994062423706,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.35213232040405273,
+        -0.09297341108322144,
+        0.27886962890625,
+        0.9998326301574707,
+        -0.015117108821868896,
+        -0.01030886173248291,
+        0.014972925186157227,
+        0.9997909069061279,
+        -0.013926804065704346,
+        0.41925930976867676,
+        0.03278625011444092,
+        0.004058837890625,
+        0.033640503883361816,
+        1.0,
+        3.993511199951172e-05,
+        2.2172927856445312e-05,
+        -3.993511199951172e-05,
+        1.0,
+        3.24249267578125e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009055018424987793,
+        0.20257973670959473,
+        0.04290628433227539,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.3511180877685547,
+        -0.10789769887924194,
+        0.3101799488067627,
+        0.9997737407684326,
+        -0.018787026405334473,
+        -0.009967148303985596,
+        0.018638968467712402,
+        0.9997174739837646,
+        -0.014750242233276367,
+        0.41925930976867676,
+        0.034955620765686035,
+        0.009042859077453613,
+        0.03505551815032959,
+        1.0,
+        -0.00013250112533569336,
+        -7.349252700805664e-05,
+        0.00013256072998046875,
+        1.0,
+        -0.00010764598846435547,
+        0.41629624366760254
+      ],
+      [
+        0.009266018867492676,
+        0.2028360366821289,
+        0.042970895767211914,
+        1.0,
+        1.1920928955078125e-07,
+        -2.980232238769531e-07,
+        -1.1920928955078125e-07,
+        1.0,
+        -1.5914440155029297e-05,
+        0.33728480339050293,
+        -0.08807200193405151,
+        0.30838990211486816,
+        0.9997389316558838,
+        -0.020735502243041992,
+        -0.009606242179870605,
+        0.020569682121276855,
+        0.9996428489685059,
+        -0.017057418823242188,
+        0.41925930976867676,
+        0.0355377197265625,
+        0.010410547256469727,
+        0.03543984889984131,
+        1.0,
+        -0.00017952919006347656,
+        -9.948015213012695e-05,
+        0.00017952919006347656,
+        1.0,
+        -0.00014573335647583008,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20256006717681885,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.3088233470916748,
+        -0.02397996187210083,
+        0.2727888822555542,
+        0.9998228549957275,
+        -0.016184329986572266,
+        -0.009604811668395996,
+        0.016006827354431152,
+        0.9997048377990723,
+        -0.01827824115753174,
+        0.41925930976867676,
+        0.03309321403503418,
+        0.003977775573730469,
+        0.03438091278076172,
+        1.0,
+        5.173683166503906e-05,
+        1.6927719116210938e-05,
+        -5.179643630981445e-05,
+        1.0,
+        4.4345855712890625e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20256006717681885,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.300417423248291,
+        0.014766693115234375,
+        0.24292731285095215,
+        0.9998749494552612,
+        -0.01231163740158081,
+        -0.009922266006469727,
+        0.01210319995880127,
+        0.9997103214263916,
+        -0.020803868770599365,
+        0.41925930976867676,
+        0.020145773887634277,
+        -0.02418738603591919,
+        0.024831175804138184,
+        0.9999992847442627,
+        0.0010178089141845703,
+        0.0006074905395507812,
+        -0.0010182857513427734,
+        0.9999990463256836,
+        0.0008230209350585938,
+        0.41629624366760254
+      ],
+      [
+        0.009091615676879883,
+        0.20256006717681885,
+        0.04294168949127197,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -9.5367431640625e-07,
+        0.31132233142852783,
+        -0.0029664039611816406,
+        0.25871026515960693,
+        0.9998986721038818,
+        -0.009363293647766113,
+        -0.010725855827331543,
+        0.009142041206359863,
+        0.9997479915618896,
+        -0.020500898361206055,
+        0.41925930976867676,
+        0.041992902755737305,
+        0.02042865753173828,
+        0.03982365131378174,
+        0.9999997615814209,
+        -0.00036710500717163086,
+        -0.00019168853759765625,
+        0.00036704540252685547,
+        0.9999998807907104,
+        -0.0003796815872192383,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20257973670959473,
+        0.042914628982543945,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.316475510597229,
+        0.06765854358673096,
+        0.2514406442642212,
+        0.9998515844345093,
+        -0.01094353199005127,
+        -0.01331174373626709,
+        0.010651946067810059,
+        0.9997060298919678,
+        -0.021778404712677002,
+        0.41925930976867676,
+        0.035611748695373535,
+        0.004885673522949219,
+        0.034360408782958984,
+        1.0,
+        -4.6253204345703125e-05,
+        -3.796815872192383e-05,
+        4.6253204345703125e-05,
+        1.0,
+        4.5299530029296875e-05,
+        0.41629624366760254
+      ],
+      [
+        0.009063482284545898,
+        0.20256006717681885,
+        0.04291260242462158,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.253512978553772,
+        0.10438930988311768,
+        0.3327815532684326,
+        0.9998939037322998,
+        -0.005053579807281494,
+        -0.01365971565246582,
+        0.0048171281814575195,
+        0.9998389482498169,
+        -0.017287731170654297,
+        0.41925930976867676,
+        0.03974413871765137,
+        0.03579390048980713,
+        0.036423683166503906,
+        0.9999992847442627,
+        -0.001103818416595459,
+        -6.61611557006836e-05,
+        0.0011037588119506836,
+        0.9999990463256836,
+        -0.0007817745208740234,
+        0.41629624366760254
+      ],
+      [
+        0.009057760238647461,
+        0.20256006717681885,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.2416396141052246,
+        0.21881604194641113,
+        0.3564988374710083,
+        0.9998669624328613,
+        -0.0066040754318237305,
+        -0.014912307262420654,
+        0.006239175796508789,
+        0.999683141708374,
+        -0.024385035037994385,
+        0.41925930976867676,
+        0.03320956230163574,
+        -0.01241910457611084,
+        0.030895113945007324,
+        0.9999996423721313,
+        0.0008132457733154297,
+        0.00021898746490478516,
+        -0.0008133649826049805,
+        0.9999994039535522,
+        0.0007567405700683594,
+        0.41629624366760254
+      ],
+      [
+        0.009069085121154785,
+        0.20257973670959473,
+        0.042914628982543945,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.1626189947128296,
+        0.28461992740631104,
+        0.3566627502441406,
+        0.9999043941497803,
+        -0.005653977394104004,
+        -0.012622654438018799,
+        0.005325913429260254,
+        0.9996511936187744,
+        -0.025870025157928467,
+        0.41925930976867676,
+        0.02411937713623047,
+        -0.018249154090881348,
+        0.03223073482513428,
+        0.9999994039535522,
+        0.0008907318115234375,
+        0.0007027387619018555,
+        -0.0008912086486816406,
+        0.9999992847442627,
+        0.0007742643356323242,
+        0.41629624366760254
+      ],
+      [
+        0.009063482284545898,
+        0.20256006717681885,
+        0.04291260242462158,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        -5.960464477539063e-08,
+        0.11943340301513672,
+        0.3110710382461548,
+        0.38945770263671875,
+        0.9999172687530518,
+        -0.004807233810424805,
+        -0.011935949325561523,
+        0.004476189613342285,
+        0.9996087551116943,
+        -0.02760899066925049,
+        0.41925930976867676,
+        0.028484582901000977,
+        -0.02769160270690918,
+        0.031121253967285156,
+        0.9999990463256836,
+        0.0012061595916748047,
+        0.0004590749740600586,
+        -0.0012065768241882324,
+        0.9999988079071045,
+        0.0009518861770629883,
+        0.41629624366760254
+      ],
+      [
+        0.008874893188476562,
+        0.20234322547912598,
+        0.04285633563995361,
+        1.0,
+        -1.1920928955078125e-07,
+        2.384185791015625e-07,
+        1.1920928955078125e-07,
+        1.0,
+        1.33514404296875e-05,
+        0.0724560022354126,
+        0.22733843326568604,
+        0.41579723358154297,
+        0.9999501705169678,
+        -0.00439983606338501,
+        -0.008969545364379883,
+        0.004159092903137207,
+        0.9996351003646851,
+        -0.02668970823287964,
+        0.41925930976867676,
+        0.03157460689544678,
+        -0.01598513126373291,
+        0.032421112060546875,
+        0.9999997615814209,
+        0.0007050037384033203,
+        0.00028765201568603516,
+        -0.0007051825523376465,
+        0.9999996423721313,
+        0.0005202293395996094,
+        0.41629624366760254
+      ],
+      [
+        0.00907754898071289,
+        0.20257973670959473,
+        0.042914628982543945,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.06938600540161133,
+        0.06680262088775635,
+        0.4239891767501831,
+        0.9999661445617676,
+        -0.0036103129386901855,
+        -0.007386624813079834,
+        0.0034453868865966797,
+        0.9997472763061523,
+        -0.022213459014892578,
+        0.41925930976867676,
+        0.032421231269836426,
+        -0.025632381439208984,
+        0.031023621559143066,
+        0.9999992847442627,
+        0.0012047290802001953,
+        0.0001195669174194336,
+        -0.0012047886848449707,
+        0.9999990463256836,
+        0.0006859302520751953,
+        0.41629624366760254
+      ],
+      [
+        0.009063482284545898,
+        0.20257973670959473,
+        0.042908430099487305,
+        1.0,
+        0.0,
+        0.0,
+        0.0,
+        1.0,
+        0.0,
+        0.03682661056518555,
+        -0.025924086570739746,
+        0.5240132808685303,
+        0.9999792575836182,
+        -0.004476070404052734,
+        -0.004615306854248047,
+        0.004358172416687012,
+        0.999671459197998,
+        -0.025255680084228516,
+        0.41925930976867676,
+        0.016828179359436035,
+        0.05714380741119385,
+        0.034651994705200195,
+        0.9999949932098389,
+        -0.003073453903198242,
+        0.0007290840148925781,
+        0.003074049949645996,
+        0.9999948740005493,
+        -0.0009063482284545898,
+        0.41629624366760254
+      ],
+      [
+        0.0065114498138427734,
+        0.20257973670959473,
+        0.042908430099487305,
+        1.0,
+        -9.5367431640625e-07,
+        2.5033950805664062e-06,
+        9.5367431640625e-07,
+        1.0,
+        -5.960464477539063e-08,
+        0.01458740234375,
+        -0.02258777618408203,
+        0.4755741357803345,
+        0.9999876022338867,
+        0.0021882057189941406,
+        -0.004497945308685303,
+        -0.0022587180137634277,
+        0.9998735189437866,
+        -0.01574110984802246,
+        0.41925930976867676,
+        0.03340005874633789,
+        0.036458492279052734,
+        0.033553123474121094,
+        0.9999995231628418,
+        -0.0007740259170532227,
+        -0.0005014538764953613,
+        0.0007735490798950195,
+        0.9999990463256836,
+        -0.0010625720024108887,
+        0.41629624366760254
+      ]
+    ]
+  ]
+}

assets/example_action_id_av_0_input.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff205f86ae169031c0e12c8c6bc8aabe24aeb44b84a16cba977669b29912f5fc
+size 1125838

assets/example_action_id_av_0_output.json

@@ -0,0 +1,669 @@
+{
+  "data": [
+    [
+      -0.004169374704360962,
+      0.0013064965605735779,
+      0.04288285970687866,
+      0.9930733442306519,
+      -0.0027953684329986572,
+      0.00307542085647583,
+      -0.002344876527786255,
+      1.0027554035186768,
+      -0.0005963500589132309
+    ],
+    [
+      -0.000341169536113739,
+      0.0040023550391197205,
+      0.030853301286697388,
+      1.0044922828674316,
+      0.0050698816776275635,
+      0.0007043546065688133,
+      -0.0014923065900802612,
+      1.0103027820587158,
+      -0.0039251744747161865
+    ],
+    [
+      0.00016222894191741943,
+      0.002112939953804016,
+      0.004467234015464783,
+      1.0028852224349976,
+      0.003892749547958374,
+      0.00044799596071243286,
+      -0.0021587014198303223,
+      1.0042486190795898,
+      -0.001470804214477539
+    ],
+    [
+      0.00045252684503793716,
+      0.003622382879257202,
+      0.0019504725933074951,
+      1.0071654319763184,
+      -0.00045464932918548584,
+      0.0014991164207458496,
+      -0.004504382610321045,
+      0.9974222183227539,
+      0.0005441531538963318
+    ],
+    [
+      -0.003174394369125366,
+      -0.0017216801643371582,
+      -0.003694474697113037,
+      1.003875494003296,
+      0.000801771879196167,
+      5.6371092796325684e-05,
+      -4.088878631591797e-05,
+      1.0089592933654785,
+      0.0010747015476226807
+    ],
+    [
+      -0.0021718740463256836,
+      0.0004750937223434448,
+      -0.0029296129941940308,
+      1.0002799034118652,
+      0.002264350652694702,
+      0.00046096742153167725,
+      -0.0009260214865207672,
+      1.00685453414917,
+      0.0006359070539474487
+    ],
+    [
+      -0.002009839750826359,
+      -0.0009101033210754395,
+      0.0028893351554870605,
+      0.9993182420730591,
+      0.0020936131477355957,
+      -1.9840896129608154e-05,
+      0.0015915334224700928,
+      1.0037269592285156,
+      -0.004174880683422089
+    ],
+    [
+      0.0034651458263397217,
+      0.0037903208285570145,
+      0.0032828450202941895,
+      1.000267505645752,
+      -0.0036886483430862427,
+      -0.0017361268401145935,
+      -0.002902016043663025,
+      0.9957513809204102,
+      0.002555273473262787
+    ],
+    [
+      0.006279021501541138,
+      0.00949704647064209,
+      0.006110593676567078,
+      1.0021371841430664,
+      -0.0037842243909835815,
+      -0.0012855827808380127,
+      -0.001484893262386322,
+      1.0039020776748657,
+      -0.00035249069333076477
+    ],
+    [
+      -0.001515701413154602,
+      -0.0002789795398712158,
+      -0.0068062953650951385,
+      1.0103508234024048,
+      0.001192629337310791,
+      0.0030622780323028564,
+      0.0011264681816101074,
+      1.007310390472412,
+      0.002111390233039856
+    ],
+    [
+      -0.0014958996325731277,
+      0.0031677186489105225,
+      -0.004236042499542236,
+      1.0023982524871826,
+      -0.002504110336303711,
+      -0.0016808435320854187,
+      0.0012012720108032227,
+      1.0030899047851562,
+      0.0005880743265151978
+    ],
+    [
+      -0.00013130903244018555,
+      0.004746794700622559,
+      0.0033633261919021606,
+      1.0057588815689087,
+      0.003953307867050171,
+      0.000896356999874115,
+      -0.0007526576519012451,
+      1.004878044128418,
+      -0.00039952993392944336
+    ],
+    [
+      0.0026078522205352783,
+      -0.0008465610444545746,
+      0.009034380316734314,
+      1.0077221393585205,
+      -0.0009244084358215332,
+      -0.0018756091594696045,
+      0.0006937459111213684,
+      1.0071992874145508,
+      0.0007385127246379852
+    ],
+    [
+      -0.0015690997242927551,
+      -0.000245044007897377,
+      0.008958447724580765,
+      1.000647783279419,
+      0.001012757420539856,
+      0.000528484582901001,
+      -0.0014475509524345398,
+      1.0043647289276123,
+      -0.0019880682229995728
+    ],
+    [
+      0.0004642903804779053,
+      0.001843869686126709,
+      0.01116645336151123,
+      1.0001274347305298,
+      0.004533201456069946,
+      0.003915667533874512,
+      0.0015954822301864624,
+      1.004312515258789,
+      4.3004751205444336e-05
+    ],
+    [
+      0.0014523789286613464,
+      -0.004249289631843567,
+      0.01856423169374466,
+      1.0013573169708252,
+      0.002036675810813904,
+      -0.001862620934844017,
+      -0.002076610689982772,
+      1.0027811527252197,
+      -0.00468212366104126
+    ],
+    [
+      -0.002909451723098755,
+      0.002633988857269287,
+      0.029188022017478943,
+      1.0077584981918335,
+      0.00022396445274353027,
+      0.00390547513961792,
+      0.0020011961460113525,
+      1.0060248374938965,
+      -0.0012506172060966492
+    ],
+    [
+      -0.0009786635637283325,
+      0.002693958580493927,
+      0.04165208339691162,
+      1.0019495487213135,
+      -0.00014898180961608887,
+      0.0024263858795166016,
+      -1.5497207641601562e-05,
+      1.010059118270874,
+      -3.0219554901123047e-05
+    ],
+    [
+      -0.0005752965807914734,
+      -0.0005559623241424561,
+      0.046234145760536194,
+      1.0054540634155273,
+      0.0020374152809381485,
+      -0.0015819594264030457,
+      -0.0029042065143585205,
+      1.0099618434906006,
+      0.000680871307849884
+    ],
+    [
+      -0.0007889866828918457,
+      -0.0012893080711364746,
+      0.059357136487960815,
+      1.000758171081543,
+      0.002288408577442169,
+      0.00029518455266952515,
+      -0.006809890270233154,
+      1.003616452217102,
+      0.0005193725228309631
+    ],
+    [
+      -0.0005115699023008347,
+      -2.2411346435546875e-05,
+      0.05575209856033325,
+      1.0020880699157715,
+      0.002775445580482483,
+      0.002016425132751465,
+      1.4990568161010742e-05,
+      1.0059070587158203,
+      0.000255754217505455
+    ],
+    [
+      -0.0032242387533187866,
+      0.0023158714175224304,
+      0.07140621542930603,
+      1.0127027034759521,
+      -0.006227180361747742,
+      0.0001257285475730896,
+      0.003853335976600647,
+      0.989827036857605,
+      -0.004152536392211914
+    ],
+    [
+      -0.004788219928741455,
+      0.001452852040529251,
+      0.0808199793100357,
+      1.0067814588546753,
+      0.0018687620759010315,
+      0.002240225672721863,
+      0.0007859393954277039,
+      1.009403109550476,
+      -0.0021954476833343506
+    ],
+    [
+      -0.00179203599691391,
+      0.0022020190954208374,
+      0.09712601453065872,
+      1.0037879943847656,
+      0.0016462206840515137,
+      -0.0021682195365428925,
+      -0.002309828996658325,
+      1.007390022277832,
+      -0.002376168966293335
+    ],
+    [
+      0.000849604606628418,
+      -0.0019690990447998047,
+      0.09653186798095703,
+      1.0044101476669312,
+      -0.00022560358047485352,
+      -0.0021284837275743484,
+      0.0007652044296264648,
+      1.0043072700500488,
+      -1.784414052963257e-05
+    ],
+    [
+      -0.0006387382745742798,
+      0.0005827993154525757,
+      0.11596892774105072,
+      1.0059268474578857,
+      0.004413425922393799,
+      -0.0023792535066604614,
+      -0.004380702972412109,
+      1.0090959072113037,
+      -0.001389428973197937
+    ],
+    [
+      -0.0010459311306476593,
+      -0.004505783319473267,
+      0.1201993077993393,
+      1.0040526390075684,
+      -0.00030914321541786194,
+      0.0003615003079175949,
+      0.004234045743942261,
+      1.0040135383605957,
+      0.005033165216445923
+    ],
+    [
+      0.002282150089740753,
+      0.005020305514335632,
+      0.134363055229187,
+      1.0035362243652344,
+      -0.007463634014129639,
+      -0.002947300672531128,
+      0.0028315000236034393,
+      1.0030007362365723,
+      0.0013876184821128845
+    ],
+    [
+      0.005076408386230469,
+      -0.0007408089004456997,
+      0.14487546682357788,
+      1.0071125030517578,
+      0.0022079087793827057,
+      -0.0019074305891990662,
+      0.0012646764516830444,
+      1.0073299407958984,
+      -0.0017407238483428955
+    ],
+    [
+      0.0033898651599884033,
+      0.0016630440950393677,
+      0.14222049713134766,
+      1.0055317878723145,
+      0.0016905665397644043,
+      -0.0016445815563201904,
+      0.0008559226989746094,
+      1.0037096738815308,
+      -0.00027045607566833496
+    ],
+    [
+      -0.00047904253005981445,
+      0.0027755796909332275,
+      0.15001334249973297,
+      1.0068082809448242,
+      -0.002282470464706421,
+      -0.0004724264144897461,
+      0.0015332624316215515,
+      0.9986047744750977,
+      -0.00040721893310546875
+    ],
+    [
+      0.0012988895177841187,
+      -0.0017725825309753418,
+      0.15345412492752075,
+      1.0003080368041992,
+      -0.004808247089385986,
+      0.0008027921430766582,
+      0.0003478825092315674,
+      1.0045523643493652,
+      -0.0006252527236938477
+    ],
+    [
+      0.0042798519134521484,
+      -0.003797680139541626,
+      0.1682387888431549,
+      1.0006740093231201,
+      0.0014480054378509521,
+      -0.004346251487731934,
+      -0.0011912062764167786,
+      1.0039736032485962,
+      0.004550337791442871
+    ],
+    [
+      0.0047804368659853935,
+      0.0020081400871276855,
+      0.18152475357055664,
+      1.0033659934997559,
+      -0.003238588571548462,
+      -0.0002244710922241211,
+      0.001187354326248169,
+      1.001799464225769,
+      0.00035965442657470703
+    ],
+    [
+      0.002194732427597046,
+      0.004285037517547607,
+      0.17377570271492004,
+      1.0024428367614746,
+      2.7358531951904297e-05,
+      0.0007259398698806763,
+      0.002799510955810547,
+      1.0095839500427246,
+      0.000986546277999878
+    ],
+    [
+      0.011655598878860474,
+      -0.004363194108009338,
+      0.1733943223953247,
+      1.0060877799987793,
+      0.005308032035827637,
+      0.00019598007202148438,
+      0.0020811930298805237,
+      1.005480170249939,
+      0.0034663397818803787
+    ],
+    [
+      0.005494013428688049,
+      0.001217433251440525,
+      0.19090485572814941,
+      1.0007939338684082,
+      -0.0017113089561462402,
+      -0.0029762238264083862,
+      -0.0029486119747161865,
+      1.007235050201416,
+      0.0011887773871421814
+    ],
+    [
+      0.009677506983280182,
+      -0.006571769714355469,
+      0.18290859460830688,
+      1.0043387413024902,
+      0.0041519105434417725,
+      -0.005755007266998291,
+      0.0021400973200798035,
+      1.0117175579071045,
+      0.0013759732246398926
+    ],
+    [
+      0.009254217147827148,
+      0.00086212158203125,
+      0.19618983566761017,
+      1.004202127456665,
+      -0.0024811476469039917,
+      -0.009780704975128174,
+      -9.959936141967773e-05,
+      1.0029432773590088,
+      0.002646505832672119
+    ],
+    [
+      0.009517982602119446,
+      0.0005913078784942627,
+      0.1987060010433197,
+      1.0014187097549438,
+      -0.003507643938064575,
+      -0.0070085227489471436,
+      0.003143489360809326,
+      1.002164363861084,
+      0.0010729804635047913
+    ],
+    [
+      0.011235184967517853,
+      -0.0012075230479240417,
+      0.1985381841659546,
+      1.00819993019104,
+      0.0004249662160873413,
+      -0.007858432829380035,
+      0.0010521560907363892,
+      1.0044105052947998,
+      0.0019039809703826904
+    ],
+    [
+      0.01611921191215515,
+      -0.001196291297674179,
+      0.2129121720790863,
+      1.0026592016220093,
+      -0.0043135881423950195,
+      -0.008859358727931976,
+      0.0018898062407970428,
+      1.0028712749481201,
+      0.001296408474445343
+    ],
+    [
+      0.012898094952106476,
+      -0.00257091224193573,
+      0.22263681888580322,
+      1.0003653764724731,
+      -0.0003224611282348633,
+      -0.006597660481929779,
+      0.001808561384677887,
+      1.0038889646530151,
+      0.0026077479124069214
+    ],
+    [
+      0.013466209173202515,
+      0.0018514543771743774,
+      0.2376641482114792,
+      1.0033774375915527,
+      -0.0018234401941299438,
+      -0.0027636736631393433,
+      -0.002201095223426819,
+      0.9944919347763062,
+      -0.003013297915458679
+    ],
+    [
+      0.016432642936706543,
+      0.0016770362854003906,
+      0.2475210428237915,
+      0.9992994070053101,
+      0.0004978589713573456,
+      -0.007409483194351196,
+      0.001736219972372055,
+      1.0018973350524902,
+      0.0006765872240066528
+    ],
+    [
+      0.014197006821632385,
+      -0.0011444091796875,
+      0.2522944509983063,
+      1.007161021232605,
+      -0.0002377331256866455,
+      -0.0076867565512657166,
+      0.00048054754734039307,
+      1.0041539669036865,
+      4.76837158203125e-07
+    ],
+    [
+      0.013951335102319717,
+      0.00016671419143676758,
+      0.26611241698265076,
+      1.003096580505371,
+      7.110834121704102e-05,
+      -0.007618337869644165,
+      -0.0012885034084320068,
+      1.0024244785308838,
+      -0.0006037205457687378
+    ],
+    [
+      0.021747827529907227,
+      -0.0007416084408760071,
+      0.2753673195838928,
+      1.004169225692749,
+      -0.0007948616985231638,
+      -0.0035905838012695312,
+      0.0017324388027191162,
+      1.0070855617523193,
+      0.0011792778968811035
+    ],
+    [
+      0.0197313129901886,
+      0.0010002106428146362,
+      0.28150367736816406,
+      1.000486135482788,
+      0.0008391812443733215,
+      -0.00998014211654663,
+      0.00013040006160736084,
+      1.0073860883712769,
+      0.00011730194091796875
+    ],
+    [
+      0.013363130390644073,
+      -0.0013455301523208618,
+      0.29323309659957886,
+      1.0011693239212036,
+      0.0004394911229610443,
+      -0.010259315371513367,
+      0.002908453345298767,
+      1.00381338596344,
+      -0.0015798062086105347
+    ],
+    [
+      0.011962294578552246,
+      0.0017336606979370117,
+      0.3051491379737854,
+      1.0071864128112793,
+      -0.00198325514793396,
+      -0.007243074476718903,
+      0.002382628619670868,
+      1.0005227327346802,
+      0.002617824822664261
+    ],
+    [
+      0.013890508562326431,
+      0.005148977041244507,
+      0.31365522742271423,
+      1.0045785903930664,
+      -0.0014152824878692627,
+      -0.0037764757871627808,
+      -1.0967254638671875e-05,
+      1.0036839246749878,
+      -0.0013008862733840942
+    ],
+    [
+      0.022371917963027954,
+      0.0010523945093154907,
+      0.32621023058891296,
+      0.9973301887512207,
+      7.641687989234924e-05,
+      -0.008195962756872177,
+      -0.0024689435958862305,
+      1.0000015497207642,
+      0.0024291127920150757
+    ],
+    [
+      0.013878557831048965,
+      0.0006828606128692627,
+      0.338645339012146,
+      0.9975997805595398,
+      -0.00040869414806365967,
+      -0.00787343829870224,
+      -0.003890112042427063,
+      1.0047005414962769,
+      0.002561509609222412
+    ],
+    [
+      0.015209555625915527,
+      0.003898248076438904,
+      0.3533927798271179,
+      0.9961752891540527,
+      -0.0017392784357070923,
+      -0.011618092656135559,
+      -0.002202928066253662,
+      1.004539132118225,
+      0.0014745891094207764
+    ],
+    [
+      0.014465630054473877,
+      0.004823088645935059,
+      0.3626636266708374,
+      1.0013577938079834,
+      -0.001902550458908081,
+      -0.007857277989387512,
+      0.0013346821069717407,
+      1.0042797327041626,
+      0.002392292022705078
+    ],
+    [
+      0.01207183301448822,
+      -0.0008889138698577881,
+      0.3779425024986267,
+      1.0077900886535645,
+      0.003589630126953125,
+      -0.006766825914382935,
+      -0.0006845667958259583,
+      1.0097190141677856,
+      0.0004222095012664795
+    ],
+    [
+      0.014950022101402283,
+      0.00266294926404953,
+      0.38445812463760376,
+      1.0014134645462036,
+      -0.0014687180519104004,
+      -0.010563522577285767,
+      -0.001019701361656189,
+      1.0056054592132568,
+      0.0019721686840057373
+    ],
+    [
+      0.017121687531471252,
+      0.0029426664113998413,
+      0.39281076192855835,
+      1.0034687519073486,
+      -0.00029963254928588867,
+      -0.014892727136611938,
+      0.00041075050830841064,
+      1.001159906387329,
+      -0.0016954541206359863
+    ],
+    [
+      0.013786494731903076,
+      0.004227876663208008,
+      0.4070053696632385,
+      1.0027498006820679,
+      0.0027508288621902466,
+      -0.006377458572387695,
+      -0.0003670156002044678,
+      0.998216986656189,
+      -0.0007525086402893066
+    ]
+  ],
+  "shape": [
+    60,
+    9
+  ],
+  "dtype": "float32"
+}

assets/example_action_id_av_1_input.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:169e65cee76ef7c6366987a8e4c8e4ec6f803f5e624ba2ef39ba1f0c672925aa
+size 1639407

assets/example_action_id_av_1_output.json

@@ -0,0 +1,669 @@
+{
+  "data": [
+    [
+      0.07009392976760864,
+      0.006693661212921143,
+      1.1420012712478638,
+      1.0078833103179932,
+      0.0012149810791015625,
+      -0.024351507425308228,
+      0.003295496106147766,
+      1.0031427145004272,
+      0.000886848196387291
+    ],
+    [
+      0.06402561813592911,
+      -0.0018425136804580688,
+      1.1193486452102661,
+      1.0022863149642944,
+      -0.003297761082649231,
+      -0.021648261696100235,
+      0.003286510705947876,
+      1.008054256439209,
+      0.0026217997074127197
+    ],
+    [
+      0.06708404421806335,
+      -0.011202313005924225,
+      1.0937659740447998,
+      1.0016952753067017,
+      -0.0032352805137634277,
+      -0.02467486262321472,
+      0.002940252423286438,
+      1.0018234252929688,
+      0.005707293748855591
+    ],
+    [
+      0.07562961429357529,
+      -0.015175968408584595,
+      1.06215238571167,
+      1.0063788890838623,
+      -0.004444316029548645,
+      -0.022849813103675842,
+      0.012309730052947998,
+      1.005750298500061,
+      0.006214454770088196
+    ],
+    [
+      0.08489534258842468,
+      -0.01856812834739685,
+      1.0337228775024414,
+      0.9988476037979126,
+      -0.006072938442230225,
+      -0.030434921383857727,
+      0.003413841128349304,
+      1.0011348724365234,
+      0.0019939541816711426
+    ],
+    [
+      0.09222845733165741,
+      -0.010918542742729187,
+      1.0194001197814941,
+      0.9980852603912354,
+      -0.0031263232231140137,
+      -0.03405023366212845,
+      0.004428897053003311,
+      1.001816987991333,
+      0.0022291988134384155
+    ],
+    [
+      0.10272128880023956,
+      0.000903397798538208,
+      1.0029981136322021,
+      0.9994226694107056,
+      -0.0014013946056365967,
+      -0.0366886705160141,
+      0.00678609311580658,
+      1.0000828504562378,
+      -0.003755345940589905
+    ],
+    [
+      0.11606456339359283,
+      0.014216836541891098,
+      0.9876351356506348,
+      0.9964910745620728,
+      -0.0012535229325294495,
+      -0.04123753681778908,
+      0.0015066340565681458,
+      1.0026605129241943,
+      -0.006498083472251892
+    ],
+    [
+      0.12204641103744507,
+      0.015004783868789673,
+      0.9837003350257874,
+      0.9974825382232666,
+      -0.0012949556112289429,
+      -0.04007109999656677,
+      0.00612206757068634,
+      0.9999562501907349,
+      -0.0036430060863494873
+    ],
+    [
+      0.12851455807685852,
+      0.015528619289398193,
+      0.9764279723167419,
+      0.9981354475021362,
+      0.0018989741802215576,
+      -0.04426932334899902,
+      0.004600256681442261,
+      0.9985778331756592,
+      -0.003377959132194519
+    ],
+    [
+      0.13196571171283722,
+      0.00893557071685791,
+      0.9678859114646912,
+      1.0045433044433594,
+      0.002807438373565674,
+      -0.051086850464344025,
+      -0.0002923011779785156,
+      1.0062892436981201,
+      -8.818507194519043e-05
+    ],
+    [
+      0.1350717842578888,
+      0.010051608085632324,
+      0.9643991589546204,
+      1.0022790431976318,
+      -0.001323610544204712,
+      -0.04617614299058914,
+      0.0016942918300628662,
+      1.0036096572875977,
+      0.0013676732778549194
+    ],
+    [
+      0.14420342445373535,
+      0.01007811725139618,
+      0.955908477306366,
+      0.9972440600395203,
+      -0.002124980092048645,
+      -0.049623310565948486,
+      -0.0002627819776535034,
+      1.000319242477417,
+      -0.0005148919299244881
+    ],
+    [
+      0.14022719860076904,
+      0.013264013454318047,
+      0.9560319185256958,
+      0.9995477199554443,
+      0.0035576671361923218,
+      -0.04792314022779465,
+      -0.001525580883026123,
+      1.003888487815857,
+      -0.0013465248048305511
+    ],
+    [
+      0.1345217525959015,
+      0.01220095157623291,
+      0.9655839204788208,
+      1.0048763751983643,
+      0.00546342134475708,
+      -0.04668119549751282,
+      -0.0010053887963294983,
+      1.0068625211715698,
+      -0.0029554516077041626
+    ],
+    [
+      0.13503174483776093,
+      0.011316493153572083,
+      0.9516907334327698,
+      1.0043013095855713,
+      0.002426639199256897,
+      -0.045573264360427856,
+      -0.0013574131298810244,
+      1.0036004781723022,
+      -0.003350198268890381
+    ],
+    [
+      0.1309308409690857,
+      0.009759962558746338,
+      0.9560006856918335,
+      0.9998754262924194,
+      0.004482567310333252,
+      -0.04438298940658569,
+      -0.001865878701210022,
+      1.0063111782073975,
+      0.0013731084764003754
+    ],
+    [
+      0.12462389469146729,
+      0.011290190741419792,
+      0.9650402069091797,
+      1.0036883354187012,
+      -0.001193612813949585,
+      -0.04106159508228302,
+      0.005374550819396973,
+      1.0081455707550049,
+      -0.0027110427618026733
+    ],
+    [
+      0.12767966091632843,
+      0.012917861342430115,
+      0.9721789360046387,
+      1.0006978511810303,
+      -0.0012182332575321198,
+      -0.04295356199145317,
+      5.793571472167969e-05,
+      1.0048669576644897,
+      -0.0009505599737167358
+    ],
+    [
+      0.12459099292755127,
+      0.016084089875221252,
+      0.9755645990371704,
+      0.99851393699646,
+      0.002149254083633423,
+      -0.040429264307022095,
+      -0.004845976829528809,
+      1.0035099983215332,
+      -0.0011209547519683838
+    ],
+    [
+      0.12318825721740723,
+      0.020346134901046753,
+      0.9927335381507874,
+      1.001643180847168,
+      0.00202333927154541,
+      -0.04205331206321716,
+      -0.004307150840759277,
+      1.0037580728530884,
+      -0.0029827356338500977
+    ],
+    [
+      0.12292107939720154,
+      0.022956013679504395,
+      1.0020689964294434,
+      1.002467393875122,
+      0.005101993680000305,
+      -0.04205179214477539,
+      -0.0018941611051559448,
+      1.0055075883865356,
+      -0.0008601397275924683
+    ],
+    [
+      0.12235790491104126,
+      0.020232800394296646,
+      1.0193688869476318,
+      0.9982290863990784,
+      0.0059588029980659485,
+      -0.03710612654685974,
+      -0.0030762851238250732,
+      1.0036814212799072,
+      0.003678947687149048
+    ],
+    [
+      0.11874179542064667,
+      0.02020188421010971,
+      1.0269538164138794,
+      1.00148344039917,
+      0.003334537148475647,
+      -0.04146520793437958,
+      -0.0018786042928695679,
+      1.0038044452667236,
+      -0.0026959776878356934
+    ],
+    [
+      0.11215314269065857,
+      0.015729710459709167,
+      1.04316246509552,
+      1.000847339630127,
+      0.002756774425506592,
+      -0.039638493210077286,
+      -0.0009539872407913208,
+      1.0064407587051392,
+      -0.0033986493945121765
+    ],
+    [
+      0.1063896119594574,
+      0.021251197904348373,
+      1.059739112854004,
+      1.0029915571212769,
+      0.008030325174331665,
+      -0.03942205011844635,
+      -0.006604194641113281,
+      1.0077309608459473,
+      -0.0016417503356933594
+    ],
+    [
+      0.10521477460861206,
+      0.018115460872650146,
+      1.071585774421692,
+      0.9995502829551697,
+      0.002011328935623169,
+      -0.03752366453409195,
+      -0.003705829381942749,
+      0.996671199798584,
+      -0.002115488052368164
+    ],
+    [
+      0.10078927874565125,
+      0.015093140304088593,
+      1.0889707803726196,
+      1.0062332153320312,
+      0.00926429033279419,
+      -0.03409150242805481,
+      -0.0007918514311313629,
+      1.0127429962158203,
+      -0.002360224723815918
+    ],
+    [
+      0.09784230589866638,
+      0.016763487830758095,
+      1.1208187341690063,
+      1.0020859241485596,
+      0.003513324074447155,
+      -0.03430382162332535,
+      0.0005292966961860657,
+      1.004651427268982,
+      -0.005986005067825317
+    ],
+    [
+      0.07926556468009949,
+      0.012953147292137146,
+      1.140628695487976,
+      0.9961432218551636,
+      0.0038596540689468384,
+      -0.030215993523597717,
+      -0.0013297051191329956,
+      1.0029637813568115,
+      -0.00047488510608673096
+    ],
+    [
+      0.0753796398639679,
+      0.00887972116470337,
+      1.1544592380523682,
+      1.0071207284927368,
+      0.004703924059867859,
+      -0.029834091663360596,
+      0.004652403295040131,
+      1.008349895477295,
+      -0.0002899765968322754
+    ],
+    [
+      0.06969903409481049,
+      0.008346021175384521,
+      1.1774840354919434,
+      1.004931926727295,
+      0.00202101469039917,
+      -0.025847388431429863,
+      0.0042855143547058105,
+      1.0110220909118652,
+      0.001279592514038086
+    ],
+    [
+      0.06848141551017761,
+      0.003982707858085632,
+      1.194669485092163,
+      0.9975674748420715,
+      0.0007158070802688599,
+      -0.02307547628879547,
+      0.0026876721531152725,
+      1.0068185329437256,
+      0.001052945852279663
+    ],
+    [
+      0.057075899094343185,
+      0.01037299633026123,
+      1.2149968147277832,
+      1.0047249794006348,
+      8.045509457588196e-05,
+      -0.02383589744567871,
+      0.006846204400062561,
+      1.007742166519165,
+      0.002701878547668457
+    ],
+    [
+      0.06148111820220947,
+      0.010166585445404053,
+      1.2267282009124756,
+      0.9979768991470337,
+      -0.0002384483814239502,
+      -0.026486635208129883,
+      0.006823986768722534,
+      1.0042164325714111,
+      -0.0020662546157836914
+    ],
+    [
+      0.054221928119659424,
+      0.01702176034450531,
+      1.2517765760421753,
+      1.0032732486724854,
+      -0.003955543041229248,
+      -0.02357548475265503,
+      0.0030149370431900024,
+      1.0078377723693848,
+      3.4650787711143494e-05
+    ],
+    [
+      0.05204963684082031,
+      0.020570974797010422,
+      1.2836378812789917,
+      0.9974656701087952,
+      -0.0004451274871826172,
+      -0.02366816997528076,
+      -0.0014014989137649536,
+      1.0038502216339111,
+      -0.0017639398574829102
+    ],
+    [
+      0.05103199928998947,
+      0.026325732469558716,
+      1.292293906211853,
+      1.001286506652832,
+      -0.0010189414024353027,
+      -0.020195096731185913,
+      0.0006793439388275146,
+      1.00400972366333,
+      -0.0025037825107574463
+    ],
+    [
+      0.04431256651878357,
+      0.021927133202552795,
+      1.3272300958633423,
+      1.0045137405395508,
+      0.000967755913734436,
+      -0.01706451177597046,
+      -3.7297606468200684e-05,
+      1.0049716234207153,
+      -0.0011880025267601013
+    ],
+    [
+      0.03555189073085785,
+      0.01882302761077881,
+      1.3371576070785522,
+      1.0066249370574951,
+      -0.0009097158908843994,
+      -0.014253586530685425,
+      -0.0018957853317260742,
+      1.0077271461486816,
+      0.0016103871166706085
+    ],
+    [
+      0.025582119822502136,
+      0.01724805310368538,
+      1.3608332872390747,
+      1.0031605958938599,
+      0.0026828348636627197,
+      -0.012003101408481598,
+      0.003360927104949951,
+      1.004669189453125,
+      6.693601608276367e-05
+    ],
+    [
+      0.01674354076385498,
+      0.012093983590602875,
+      1.3856382369995117,
+      1.0011014938354492,
+      -0.0013251304626464844,
+      -0.011342525482177734,
+      0.0018950244411826134,
+      1.0027016401290894,
+      0.0013082325458526611
+    ],
+    [
+      0.015420682728290558,
+      0.015567049384117126,
+      1.4040814638137817,
+      0.9977735877037048,
+      0.0011159181594848633,
+      -0.009455356746912003,
+      -3.6954879760742188e-06,
+      1.0064294338226318,
+      0.001549556851387024
+    ],
+    [
+      0.010074913501739502,
+      0.015535108745098114,
+      1.4403393268585205,
+      1.0040385723114014,
+      -0.0023125112056732178,
+      -0.0028183162212371826,
+      0.0014582127332687378,
+      1.0049704313278198,
+      6.872415542602539e-05
+    ],
+    [
+      0.008311420679092407,
+      0.017948955297470093,
+      1.4601424932479858,
+      0.9989641308784485,
+      0.001294134184718132,
+      -0.005496010184288025,
+      0.0015567168593406677,
+      1.005094051361084,
+      -0.0007513612508773804
+    ],
+    [
+      0.0035017579793930054,
+      0.020573198795318604,
+      1.4812562465667725,
+      1.0009287595748901,
+      0.004388183355331421,
+      -0.0034145116806030273,
+      0.0032011419534683228,
+      1.0060547590255737,
+      0.0006353855133056641
+    ],
+    [
+      0.0016254633665084839,
+      0.025684982538223267,
+      1.5020363330841064,
+      1.0030529499053955,
+      0.0016576647758483887,
+      -0.005533203482627869,
+      0.006280124187469482,
+      1.0066683292388916,
+      -0.004450388252735138
+    ],
+    [
+      0.004891932010650635,
+      0.022001147270202637,
+      1.5159921646118164,
+      0.9982910752296448,
+      0.0019840586464852095,
+      0.0025368332862854004,
+      0.003538757562637329,
+      1.0018901824951172,
+      -0.002766549587249756
+    ],
+    [
+      0.005643934011459351,
+      0.017889663577079773,
+      1.5284485816955566,
+      0.9957029223442078,
+      0.0005564317107200623,
+      -0.005682408809661865,
+      0.004203110933303833,
+      1.0006530284881592,
+      -0.002186410129070282
+    ],
+    [
+      0.0013659894466400146,
+      0.017303138971328735,
+      1.5459176301956177,
+      0.9986610412597656,
+      0.0025620944797992706,
+      -0.0046241506934165955,
+      0.0023546814918518066,
+      1.0049318075180054,
+      -0.0027633383870124817
+    ],
+    [
+      0.0020876526832580566,
+      0.01876453310251236,
+      1.5635592937469482,
+      1.0048083066940308,
+      0.002034813165664673,
+      -0.00023903697729110718,
+      0.0019740089774131775,
+      1.0029975175857544,
+      -0.0030378326773643494
+    ],
+    [
+      -0.000969788758084178,
+      0.02315378189086914,
+      1.5827395915985107,
+      1.0053610801696777,
+      0.0017460882663726807,
+      0.0014020651578903198,
+      0.005061998963356018,
+      1.0060739517211914,
+      -0.001481384038925171
+    ],
+    [
+      -0.0009270310401916504,
+      0.021047085523605347,
+      1.591257095336914,
+      0.9957737326622009,
+      0.000979650765657425,
+      -0.00326402485370636,
+      0.0065198540687561035,
+      1.0065511465072632,
+      -0.004033505916595459
+    ],
+    [
+      0.0006775943329557776,
+      0.021081745624542236,
+      1.6065882444381714,
+      0.9954938888549805,
+      -0.00036662817001342773,
+      -0.0016272813081741333,
+      0.0027448534965515137,
+      1.0027532577514648,
+      -0.0011054277420043945
+    ],
+    [
+      0.0006171315908432007,
+      0.023595139384269714,
+      1.63612699508667,
+      0.9980154037475586,
+      0.0018391646444797516,
+      -0.0013407617807388306,
+      0.0035662055015563965,
+      1.0019949674606323,
+      0.0002149343490600586
+    ],
+    [
+      0.00248805433511734,
+      0.024855956435203552,
+      1.6411882638931274,
+      1.003147840499878,
+      -0.0025542080402374268,
+      -0.0012470632791519165,
+      0.0017508864402770996,
+      1.0075249671936035,
+      0.004809528589248657
+    ],
+    [
+      0.004023965448141098,
+      0.02341434359550476,
+      1.6552321910858154,
+      0.9999925494194031,
+      0.003260374069213867,
+      -0.004240959882736206,
+      -0.0003981068730354309,
+      1.0055534839630127,
+      -0.002688378095626831
+    ],
+    [
+      0.005011148750782013,
+      0.021908849477767944,
+      1.6845402717590332,
+      0.9952720403671265,
+      0.0013539791107177734,
+      -0.004400491714477539,
+      -0.0024556964635849,
+      1.0060759782791138,
+      0.0018045902252197266
+    ],
+    [
+      0.00039759278297424316,
+      0.022936120629310608,
+      1.6785025596618652,
+      0.9988229274749756,
+      0.0002734065055847168,
+      -0.006101280450820923,
+      -0.00011269748210906982,
+      1.0011308193206787,
+      -0.0015612244606018066
+    ],
+    [
+      -0.00012701749801635742,
+      0.022358208894729614,
+      1.6973373889923096,
+      1.0032126903533936,
+      0.001924470067024231,
+      -0.0016700327396392822,
+      0.0007153153419494629,
+      1.0081400871276855,
+      0.0002083033323287964
+    ]
+  ],
+  "shape": [
+    60,
+    9
+  ],
+  "dtype": "float32"
+}

assets/example_i2v_output.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dce2d76053d4283342bb4103a3a32a28b5529a97b15cacc386dd75c812601779
+size 17298256

assets/example_i2v_prompt.json

@@ -0,0 +1,124 @@
+{
+    "subjects": [
+      {
+        "description": "A car (viewed from the driver's perspective/dashcam) traveling along a coastal mountain road. The vehicle's grey dashboard and windshield frame are visible at the bottom of the frame.",
+        "appearance_details": "Grey/dark interior dashboard, windshield wipers visible, clean windshield providing clear view of the road ahead",
+        "relationship": "Primary subject and point-of-view vehicle navigating the mountain road",
+        "location": "Bottom foreground, providing the POV perspective",
+        "relative_size": "Large within frame (dashboard occupies bottom portion)",
+        "orientation": "Facing forward along the road direction",
+        "pose": "",
+        "action": "Driving fast along the curved coastal road, then making a sudden emergency stop",
+        "state_changes": "Transitions from fast forward motion to abrupt deceleration and complete stop",
+        "clothing": "",
+        "expression": "",
+        "gender": "",
+        "age": "",
+        "skin_tone_and_texture": "",
+        "facial_features": "",
+        "number_of_subjects": 1,
+        "number_of_arms": 0,
+        "number_of_legs": 0
+      },
+      {
+        "description": "Falling rocks and debris from the steep rocky mountain cliff face, creating a landslide that tumbles down toward the road surface",
+        "appearance_details": "Grey and brown angular rocks of various sizes, accompanied by dust and smaller debris, mixed with displaced green vegetation",
+        "relationship": "Obstacle that forces the car to make an emergency stop",
+        "location": "Center and center-right of frame, falling from the mountain cliff face toward the road",
+        "relative_size": "Medium within frame, growing larger as rocks approach the road",
+        "orientation": "Falling downward from the cliff face",
+        "pose": "",
+        "action": "Tumbling and cascading down the steep rocky mountainside toward the road",
+        "state_changes": "Begins as small movement on the cliff face, escalates into a significant rockfall with dust cloud",
+        "clothing": "",
+        "expression": "",
+        "gender": "",
+        "age": "",
+        "skin_tone_and_texture": "",
+        "facial_features": "",
+        "number_of_subjects": 0,
+        "number_of_arms": 0,
+        "number_of_legs": 0
+      }
+    ],
+    "background_setting": "A dramatic coastal mountain highway carved along steep rocky cliffs. The road is a well-maintained two-lane highway with white dashed center lines, solid white edge lines, and raised reflective road markers. A metal guardrail runs along the right side protecting from a drop toward the ocean. The mountain rises steeply on the left with exposed grey and brown rock faces partially covered with lush green native vegetation including ferns and shrubs. The ocean is visible in the distance to the right under a partly cloudy blue sky. The road curves sharply to the left around the mountain headland.",
+    "lighting": {
+      "conditions": "Bright natural daylight with partly cloudy skies, strong ambient illumination",
+      "direction": "Top-lit and slightly front-lit from the sun positioned high in the sky",
+      "shadows": "Moderate shadows cast by the mountain cliff creating areas of shade on the road and vegetation, soft shadows from clouds",
+      "illumination_effect": "Bright, clear visibility with good contrast between the sunlit road surface and shaded mountain areas, natural coastal light"
+    },
+    "aesthetics": {
+      "composition": "Driver's POV/dashcam perspective with the road curving ahead as the central leading line, mountain dominating the upper-left portion, ocean visible to the right, dashboard framing the bottom",
+      "color_scheme": "Dominant greens from vegetation, grey-brown rocky cliff faces, dark grey asphalt road with white markings, blue sky and ocean, yellow warning sign providing accent color",
+      "mood_atmosphere": "Initially scenic and adventurous, transitioning to tense and dangerous as the landslide occurs",
+      "patterns": "Repeating metal guardrail posts along the right side, dashed center line markings, layered rock strata on the cliff face"
+    },
+    "cinematography": {
+      "camera_motion": "Forward-moving dashcam perspective with slight vibration from vehicle speed, sudden forward lurch during emergency braking",
+      "framing": "Wide shot from driver's POV through windshield",
+      "camera_angle": "Eye-level from driver's seated position",
+      "depth_of_field": "Deep",
+      "focus": "Road ahead and mountain cliff face in sharp focus",
+      "lens_focal_length": "Wide-angle (approximately 28-35mm equivalent)"
+    },
+    "style_medium": "Live-action video",
+    "artistic_style": "Realistic dashcam footage, dramatic documentary style",
+    "context": "Dashcam footage capturing a dangerous landslide event on a coastal mountain highway, requiring emergency driving maneuver",
+    "actions": [
+      {
+        "time": "0:00-0:03",
+        "description": "The car drives fast along the winding coastal mountain road, approaching the left curve with the road rushing beneath"
+      },
+      {
+        "time": "0:03-0:05",
+        "description": "Rocks begin to dislodge from the steep cliff face ahead, tumbling down with increasing intensity as a landslide develops, dust rising from impacts"
+      },
+      {
+        "time": "0:05-0:07",
+        "description": "The car makes a sudden emergency stop, the camera lurching forward from the deceleration as rocks and debris continue falling onto the road ahead"
+      }
+    ],
+    "text_and_signage_elements": [
+      {
+        "text": "Left curve arrow",
+        "category": "scene_sign",
+        "appearance": "Black arrow on yellow diamond-shaped warning sign, mounted on a metal pole",
+        "spatial_temporal": "Left side of road, visible from 0:00 and passing by as car moves forward",
+        "context": "Road warning sign indicating a sharp left curve ahead, standard highway signage"
+      }
+    ],
+    "segments": [
+      {
+        "segment_index": 0,
+        "time_range": "0:00-0:03",
+        "description": "The car speeds along the scenic coastal mountain highway, navigating the curve. The road surface rushes beneath, white lines blur slightly from speed, and the mountain looms ahead.",
+        "key_changes": "Road perspective shifts as car navigates the curve, passing the warning sign, guardrail moves in peripheral view",
+        "camera": "Forward-moving dashcam with slight road vibration, steady forward momentum"
+      },
+      {
+        "segment_index": 1,
+        "time_range": "0:03-0:05",
+        "description": "Rocks begin breaking loose from the cliff face ahead. Small stones first, then larger boulders cascade down the mountainside creating a growing dust cloud and debris field on and near the road.",
+        "key_changes": "Mountain transitions from static backdrop to active hazard, dust cloud forms, debris appears on road surface ahead",
+        "camera": "Still moving forward but beginning to slow, slight camera shake increases"
+      },
+      {
+        "segment_index": 2,
+        "time_range": "0:05-0:07",
+        "description": "The car performs an emergency stop. The camera pitches forward from sudden braking force. The vehicle comes to a complete halt with the landslide debris visible on the road ahead, dust still settling.",
+        "key_changes": "Rapid deceleration, camera lurches forward then settles, motion stops completely, dust and small rocks still visible falling",
+        "camera": "Abrupt forward pitch from braking, then stabilizes to static as vehicle stops"
+      }
+    ],
+    "transitions": [],
+    "temporal_caption": "The video begins with a dashcam view of a car driving fast along a scenic coastal mountain road, the asphalt rushing beneath and a sharp left curve sign visible on the left. The car navigates the curve at speed with the steep rocky cliff towering above and the ocean visible to the right. Around the 3-second mark, small rocks begin to dislodge from the cliff face ahead, quickly escalating into a significant rockfall with larger boulders tumbling down the mountainside and a dust cloud forming. By second 5, the driver reacts with emergency braking - the camera lurches forward dramatically from the sudden deceleration. The car comes to a complete stop by second 6-7, with the road ahead partially blocked by fallen rocks and debris, dust still settling in the air around the landslide zone.",
+    "audio_description": "Initially the sound of a car engine at moderate-high RPM and tire noise on asphalt with wind passing over the vehicle. Around second 3, the rumbling and cracking sounds of rocks breaking loose begin, growing louder with impacts of stones hitting the road surface and each other. Loud thuds and crashes as larger rocks land. At second 5, the sharp screech of tires braking hard on asphalt, followed by the settling sounds of smaller pebbles and dust, with distant rumbling of the remaining rockfall subsiding.",
+    "resolution": {
+      "W": 1280,
+      "H": 720
+    },
+    "aspect_ratio": "16,9",
+    "duration": "7s",
+    "fps": 24
+  }

assets/example_i2vs_output.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:421401e0d16252575c042512ed9f6e3b296fe239e71ee861bc693ed542534d46
+size 15752836

assets/example_reasoning_prompt.json

@@ -0,0 +1,4 @@
+{
+  "prompt": "The task is to put flower into the red bottle. Generate a plan consisting of subtasks for accomplish the task.",
+  "max_tokens": 4096
+}

assets/example_t2v_diffusers_output.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d59ff3a81c442dba15951dca10876e0df855ca054503a2e905cf63996b852327
+size 1693143

assets/example_t2v_output.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f235b9e0cbc8fe455e3d730a7e89e861be546e9d240e86515f4bece4d60334e
+size 9514932

assets/example_t2v_prompt.json

@@ -0,0 +1,115 @@
+{
+    "subjects": [
+      {
+        "description": "A modern industrial robotic arm with a silver and dark gray metallic body, featuring multiple articulated joints and a flat rubber-padded gripper end-effector holding a green sponge. The arm has visible hydraulic cables and a smooth polished finish.",
+        "appearance_details": "The robotic arm has a sturdy base bolted to the countertop, with branded serial number markings on the shoulder joint. The gripper holds a standard green and yellow kitchen sponge. Small LED indicator lights glow blue near the base.",
+        "relationship": "Primary subject interacting with the dirty plate",
+        "location": "Center-right of frame",
+        "relative_size": "Large within frame",
+        "orientation": "Angled toward camera at roughly 45 degrees, arm extending downward toward the plate",
+        "pose": "Extended downward with the end-effector pressing the sponge against the plate surface",
+        "action": "Wiping a dirty plate with circular and sweeping motions",
+        "state_changes": "The arm moves fluidly from one side of the plate to the other, rotating its wrist joint to apply even pressure across the plate surface",
+        "clothing": "",
+        "expression": "",
+        "gender": "",
+        "age": "",
+        "skin_tone_and_texture": "",
+        "facial_features": "",
+        "number_of_subjects": 1,
+        "number_of_arms": 0,
+        "number_of_legs": 0
+      },
+      {
+        "description": "A white ceramic dinner plate with dried food residue, sauce stains, and grease marks covering its surface. Standard 10-inch round plate with a slightly raised rim.",
+        "appearance_details": "The plate has brownish-orange dried sauce, bits of dried food particles, and oily smears. As the sponge wipes across, clean white ceramic is revealed underneath.",
+        "relationship": "Object being cleaned by the robotic arm",
+        "location": "Center-left foreground, resting on the countertop",
+        "relative_size": "Medium within frame",
+        "orientation": "Flat, face-up on the counter",
+        "pose": "Stationary on the kitchen countertop",
+        "action": "Being wiped clean",
+        "state_changes": "Progressively becomes cleaner as the robotic arm wipes away the food residue, transitioning from dirty to mostly clean",
+        "clothing": "",
+        "expression": "",
+        "gender": "",
+        "age": "",
+        "skin_tone_and_texture": "",
+        "facial_features": "",
+        "number_of_subjects": 1,
+        "number_of_arms": 0,
+        "number_of_legs": 0
+      }
+    ],
+    "background_setting": "A modern, well-organized residential kitchen with light gray granite countertops, white cabinetry with brushed nickel handles, a stainless steel sink visible to the left, and a tiled backsplash in a subtle herringbone pattern. A dish rack with a few clean plates sits near the sink. A window above the sink lets in natural daylight. Small potted herbs sit on the windowsill.",
+    "lighting": {
+      "conditions": "Bright, mixed natural and artificial lighting \u2014 daylight from the window supplemented by warm overhead LED kitchen lights",
+      "direction": "Primary light from the left (window) with soft overhead fill from above",
+      "shadows": "Soft shadows cast by the robotic arm onto the countertop and plate, with a gentle shadow beneath the plate",
+      "illumination_effect": "Clean, well-lit domestic atmosphere with slight warm tones from the overhead lights blending with cooler daylight from the window"
+    },
+    "aesthetics": {
+      "composition": "The robotic arm and plate are centered in the frame with the kitchen environment providing context in the background. The diagonal line of the arm creates dynamic visual interest.",
+      "color_scheme": "Neutral palette of whites, grays, and silver metallics with pops of green from the sponge and herbs, warm wood tones from a cutting board in the background",
+      "mood_atmosphere": "Futuristic domestic, clean, efficient, slightly whimsical",
+      "patterns": "Herringbone tile pattern on the backsplash"
+    },
+    "cinematography": {
+      "camera_motion": "Slow, subtle push-in toward the plate as it gets cleaner",
+      "framing": "Medium close-up shot capturing the robotic arm, plate, and immediate countertop area",
+      "camera_angle": "Slightly high angle, approximately 30 degrees above eye level, looking down at the plate",
+      "depth_of_field": "Shallow",
+      "focus": "Sharp focus on the sponge-plate contact point and the robotic arm's gripper",
+      "lens_focal_length": "50mm equivalent"
+    },
+    "style_medium": "Live-action video",
+    "artistic_style": "Realistic, clean tech-demo aesthetic with cinematic color grading",
+    "context": "Demonstration of a robotic kitchen assistant performing a household chore, suitable for a technology showcase or smart home advertisement",
+    "actions": [
+      {
+        "time": "0:00-0:02",
+        "description": "The robotic arm lowers the sponge onto the dirty plate and begins its first wiping pass from the center outward"
+      },
+      {
+        "time": "0:02-0:05",
+        "description": "The arm performs circular wiping motions across the plate surface, rotating its wrist joint, progressively removing dried food and sauce stains"
+      },
+      {
+        "time": "0:05-0:07",
+        "description": "The arm makes final sweeping passes across the now mostly-clean plate, then lifts the sponge slightly, revealing the cleaned surface"
+      }
+    ],
+    "text_and_signage_elements": [],
+    "segments": [
+      {
+        "segment_index": 0,
+        "time_range": "0:00-0:02",
+        "description": "The robotic arm descends and makes initial contact with the dirty plate, beginning the wiping process",
+        "key_changes": "Arm transitions from hovering position to active contact with plate; first streak of clean ceramic appears",
+        "camera": "Static with very subtle push-in beginning"
+      },
+      {
+        "segment_index": 1,
+        "time_range": "0:02-0:05",
+        "description": "Main cleaning action as the robotic arm performs systematic circular wiping motions across the plate",
+        "key_changes": "Plate progressively becomes cleaner; food residue is visibly displaced by the sponge; water droplets and suds appear",
+        "camera": "Continuing slow push-in, maintaining focus on the cleaning action"
+      },
+      {
+        "segment_index": 2,
+        "time_range": "0:05-0:07",
+        "description": "Final cleaning passes and the arm lifts away to reveal the clean plate",
+        "key_changes": "Plate transitions from mostly clean to fully clean; arm lifts sponge and retracts slightly upward",
+        "camera": "Push-in completes; camera holds steady on the clean plate"
+      }
+    ],
+    "transitions": [],
+    "temporal_caption": "The video opens with a robotic arm positioned above a dirty white plate on a kitchen countertop. In the first two seconds, the arm lowers its sponge-equipped gripper onto the plate and begins a sweeping motion from center to edge. From seconds two through five, the arm performs methodical circular wiping motions, its wrist joint rotating smoothly as dried food and sauce stains are progressively removed, revealing clean white ceramic beneath. Small water droplets and light suds form on the plate surface. In the final two seconds, the arm completes its last passes across the now-clean plate and lifts the sponge upward, pausing briefly as if inspecting its work, with the gleaming clean plate fully visible below.",
+    "resolution": {
+      "W": 1280,
+      "H": 720
+    },
+    "aspect_ratio": "16,9",
+    "duration": "7s",
+    "fps": 24
+  }

assets/example_t2v_prompt_short.txt

@@ -0,0 +1 @@
+A robot arm is cleaning a plate in the kitchen

assets/example_t2vs_output.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:929bb8ea9a4da34da334c9cedfd696bb89a80b9f018ce732a586143be45331f0
+size 4407722

assets/example_t2vs_prompt.json

@@ -0,0 +1,136 @@
+{
+    "subjects": [
+      {
+        "description": "A sleek industrial robot arm with a silver and dark gray metallic finish, featuring multiple articulated joints and a gripper end-effector that holds a clear glass jar filled with water.",
+        "appearance_details": "The robot arm has visible servo motors at each joint, subtle branding embossed on the upper arm segment, black rubber gaskets at joint connections, and a precision two-finger parallel gripper clamping the jar securely.",
+        "relationship": "Primary actor performing the pouring action; positioned above and beside the cup.",
+        "location": "Center-right of frame",
+        "relative_size": "Large within frame",
+        "orientation": "Angled toward the left side of frame, gripper tilted downward toward the cup",
+        "pose": "Extended arm with elbow joint bent, wrist rotated to tilt the jar at a pouring angle",
+        "action": "Pouring water from a glass jar into a cup",
+        "state_changes": "Begins in a neutral upright hold, then gradually tilts the jar to pour, and returns slightly upright as the pour finishes.",
+        "clothing": "",
+        "expression": "",
+        "gender": "",
+        "age": "",
+        "skin_tone_and_texture": "",
+        "facial_features": "",
+        "number_of_subjects": 1,
+        "number_of_arms": 0,
+        "number_of_legs": 0
+      },
+      {
+        "description": "A clear glass jar approximately three-quarters full of water, with a wide mouth opening, held by the robot arm's gripper.",
+        "appearance_details": "Transparent glass with slight green tint, smooth cylindrical body, no label, water visible sloshing gently inside as the jar tilts.",
+        "relationship": "Held by the robot arm; source of the water being poured.",
+        "location": "Center-right, elevated above the cup",
+        "relative_size": "Medium within frame",
+        "orientation": "Tilting progressively toward the left as the pour occurs",
+        "pose": "Held at an angle by the gripper",
+        "action": "Being tilted to pour water",
+        "state_changes": "Water level decreases as the pour progresses; jar tilts from near-vertical to approximately 45 degrees.",
+        "clothing": "",
+        "expression": "",
+        "gender": "",
+        "age": "",
+        "skin_tone_and_texture": "",
+        "facial_features": "",
+        "number_of_subjects": 1,
+        "number_of_arms": 0,
+        "number_of_legs": 0
+      },
+      {
+        "description": "A white ceramic cup sitting on a flat surface, positioned to receive the poured water.",
+        "appearance_details": "Simple cylindrical mug shape with a small handle on the right side, matte white glaze, clean and empty at the start.",
+        "relationship": "Receiving vessel for the water being poured from the jar.",
+        "location": "Center-left foreground, on the table surface",
+        "relative_size": "Small within frame",
+        "orientation": "Upright, opening facing upward",
+        "pose": "Stationary on the table",
+        "action": "Receiving water",
+        "state_changes": "Gradually fills with water as the pour continues.",
+        "clothing": "",
+        "expression": "",
+        "gender": "",
+        "age": "",
+        "skin_tone_and_texture": "",
+        "facial_features": "",
+        "number_of_subjects": 1,
+        "number_of_arms": 0,
+        "number_of_legs": 0
+      }
+    ],
+    "background_setting": "A clean, minimalist laboratory or workshop environment with a plain light gray wall in the background and a smooth white tabletop surface. The space is uncluttered, with no other objects or distractions visible, contributing to a very quiet and sterile atmosphere. The edges of the table are just visible at the bottom of the frame.",
+    "lighting": {
+      "conditions": "Soft, even studio lighting with minimal harsh highlights, creating a clean and controlled look.",
+      "direction": "Front-lit with slight top-down component, providing even illumination across the robot arm and objects.",
+      "shadows": "Soft, diffused shadows beneath the cup and the robot arm's base, with gentle shadow on the table surface.",
+      "illumination_effect": "The lighting emphasizes the metallic sheen of the robot arm and the transparency of the water and glass jar, giving the scene a polished, technical demonstration feel."
+    },
+    "aesthetics": {
+      "composition": "The robot arm dominates the right half of the frame while the cup sits in the left-center foreground, creating a clear visual flow from right to left following the pouring action. Negative space above and behind keeps focus on the action.",
+      "color_scheme": "Neutral palette dominated by silver, gray, and white with the clear blue-tinted transparency of water providing subtle color contrast.",
+      "mood_atmosphere": "Calm, precise, clinical, quietly impressive",
+      "patterns": ""
+    },
+    "cinematography": {
+      "camera_motion": "Static",
+      "framing": "Medium shot",
+      "camera_angle": "Eye-level, slightly elevated",
+      "depth_of_field": "Shallow",
+      "focus": "Sharp focus on the pouring point where water exits the jar and enters the cup",
+      "lens_focal_length": "50mm equivalent"
+    },
+    "style_medium": "Live-action video",
+    "artistic_style": "Realistic, clean technical demonstration",
+    "context": "A robotics demonstration showcasing a robot arm's precision and dexterity in performing a delicate pouring task.",
+    "actions": [
+      {
+        "time": "0:00-0:02",
+        "description": "The robot arm holds the jar upright in a steady position above the cup, making small preparatory adjustments to align the jar's mouth with the cup opening."
+      },
+      {
+        "time": "0:02-0:05",
+        "description": "The robot arm smoothly tilts the jar, and a steady stream of clear water flows from the jar into the white ceramic cup below. The water stream is smooth and controlled."
+      },
+      {
+        "time": "0:05-0:07",
+        "description": "The robot arm gradually returns the jar to a more upright position, the water stream thins and stops, and the cup is now partially filled with water. The arm holds still in its final position."
+      }
+    ],
+    "text_and_signage_elements": [],
+    "segments": [
+      {
+        "segment_index": 0,
+        "time_range": "0:00-0:02",
+        "description": "Opening shot establishes the robot arm holding the jar above the cup. The arm makes slight positional adjustments.",
+        "key_changes": "Minor wrist rotation as the arm aligns the jar with the cup.",
+        "camera": "Static, medium shot at eye level."
+      },
+      {
+        "segment_index": 1,
+        "time_range": "0:02-0:05",
+        "description": "The main pouring action occurs as the robot arm tilts the jar and water flows in a controlled stream into the cup.",
+        "key_changes": "Jar tilts from near-vertical to approximately 45 degrees; water stream begins and maintains a steady flow; cup fills progressively.",
+        "camera": "Static, maintaining focus on the pouring point."
+      },
+      {
+        "segment_index": 2,
+        "time_range": "0:05-0:07",
+        "description": "The pour concludes as the arm returns the jar upright. The water stream tapers off and the scene settles into stillness.",
+        "key_changes": "Jar returns toward vertical; water stream narrows and ceases; cup now holds water; arm stabilizes.",
+        "camera": "Static, same framing held throughout."
+      }
+    ],
+    "transitions": [],
+    "temporal_caption": "The video opens with a silver robotic arm holding a clear glass jar of water above a white ceramic cup on a clean white table against a gray background. During the first two seconds, the arm makes subtle alignment adjustments. At around two seconds, the wrist joint rotates and the jar begins to tilt, releasing a smooth, steady stream of clear water that arcs downward into the cup. The pouring continues for about three seconds, with the water level in the jar visibly decreasing and the cup gradually filling. Around the five-second mark, the arm begins to level the jar back upright, the stream of water thins to a trickle, then stops entirely. For the remaining two seconds, the robot arm holds the jar still in a slightly tilted resting position, and the filled cup sits motionless on the table.",
+    "audio_description": "The scene is very quiet. The dominant sound is the gentle splashing and trickling of water as it pours from the jar into the cup, starting softly and becoming slightly louder as the cup fills. There is a faint mechanical whir from the robot arm's servo motors during movement. No music, no speech, and minimal ambient noise, emphasizing the tranquility of the environment.",
+    "resolution": {
+      "H": 720,
+      "W": 1280
+    },
+    "aspect_ratio": "16,9",
+    "duration": "7s",
+    "fps": 24
+  }

assets/negative_prompt.json

@@ -0,0 +1,108 @@
+{
+  "subjects": [
+    {
+      "description": "Blurry, poorly defined subjects with inconsistent shapes and unrealistic proportions.",
+      "appearance_details": "Distorted features, visible compression artifacts, muddy textures lacking fine detail, color bleeding between elements, and unnatural skin tones or surface textures that appear artificial or computer-generated.",
+      "relationship": "Subjects appear disconnected from the environment, floating or improperly grounded in the scene without proper occlusion or spatial coherence.",
+      "location": "Subjects are poorly placed within the frame, appearing at awkward positions that violate basic compositional rules.",
+      "relative_size": "Inconsistent scale relationships between subjects and the environment, with objects appearing too large or too small relative to their surroundings.",
+      "orientation": "Unnatural orientations that defy physics and spatial logic.",
+      "pose": "Stiff, mannequin-like poses with unnatural joint angles and impossible limb positions that look computer-generated.",
+      "action": "Incoherent motion with visible frame-to-frame discontinuities. Movement appears as a slideshow rather than smooth animation. Limbs and appendages pop between positions without interpolation.",
+      "state_changes": "Visual state transitions are abrupt and jarring. Colors shift without motivation. Surface textures flicker between different materials randomly. Outlines shimmer and vibrate.",
+      "clothing": "Clothing appears painted on with no sense of material weight or drape. Fabric textures are flat and repeat visibly.",
+      "expression": "Frozen, uncanny valley expressions or expressions that change abruptly without natural transition.",
+      "gender": "",
+      "age": "",
+      "skin_tone_and_texture": "Waxy, plastic-looking skin with visible artifacts and inconsistent texture resolution across the frame.",
+      "facial_features": "Asymmetric facial features, extra fingers or limbs, teeth that appear blurry or malformed.",
+      "number_of_subjects": 0,
+      "number_of_arms": 0,
+      "number_of_legs": 0
+    },
+    {
+      "description": "Extremely low-quality subjects with visible rendering artifacts, broken mesh geometry, and completely unrealistic proportions throughout.",
+      "appearance_details": "Distorted features, visible compression artifacts, muddy textures lacking fine detail, color bleeding between elements, and unnatural skin tones or surface textures that appear artificial or computer-generated.",
+      "relationship": "Subjects appear disconnected from the environment, floating or improperly grounded in the scene without proper occlusion or spatial coherence.",
+      "location": "Subjects are poorly placed within the frame, appearing at awkward positions that violate basic compositional rules.",
+      "relative_size": "Inconsistent scale relationships between subjects and the environment, with objects appearing too large or too small relative to their surroundings.",
+      "orientation": "Unnatural orientations that defy physics and spatial logic.",
+      "pose": "Stiff, mannequin-like poses with unnatural joint angles and impossible limb positions that look computer-generated.",
+      "action": "Incoherent motion with visible frame-to-frame discontinuities. Movement appears as a slideshow rather than smooth animation. Limbs and appendages pop between positions without interpolation.",
+      "state_changes": "Visual state transitions are abrupt and jarring. Colors shift without motivation. Surface textures flicker between different materials randomly. Outlines shimmer and vibrate.",
+      "clothing": "Clothing appears painted on with no sense of material weight or drape. Fabric textures are flat and repeat visibly.",
+      "expression": "Frozen, uncanny valley expressions or expressions that change abruptly without natural transition.",
+      "gender": "",
+      "age": "",
+      "skin_tone_and_texture": "Waxy, plastic-looking skin with visible artifacts and inconsistent texture resolution across the frame.",
+      "facial_features": "Asymmetric facial features, extra fingers or limbs, teeth that appear blurry or malformed.",
+      "number_of_subjects": 0,
+      "number_of_arms": 0,
+      "number_of_legs": 0
+    },
+    {
+      "description": "Poorly generated subjects exhibiting all hallmarks of failed neural rendering \u2014 flickering edges, inconsistent depth, and uncanny spatial relationships.",
+      "appearance_details": "Distorted features, visible compression artifacts, muddy textures lacking fine detail, color bleeding between elements, and unnatural skin tones or surface textures that appear artificial or computer-generated.",
+      "relationship": "Subjects appear disconnected from the environment, floating or improperly grounded in the scene without proper occlusion or spatial coherence.",
+      "location": "Subjects are poorly placed within the frame, appearing at awkward positions that violate basic compositional rules.",
+      "relative_size": "Inconsistent scale relationships between subjects and the environment, with objects appearing too large or too small relative to their surroundings.",
+      "orientation": "Unnatural orientations that defy physics and spatial logic.",
+      "pose": "Stiff, mannequin-like poses with unnatural joint angles and impossible limb positions that look computer-generated.",
+      "action": "Incoherent motion with visible frame-to-frame discontinuities. Movement appears as a slideshow rather than smooth animation. Limbs and appendages pop between positions without interpolation.",
+      "state_changes": "Visual state transitions are abrupt and jarring. Colors shift without motivation. Surface textures flicker between different materials randomly. Outlines shimmer and vibrate.",
+      "clothing": "Clothing appears painted on with no sense of material weight or drape. Fabric textures are flat and repeat visibly.",
+      "expression": "Frozen, uncanny valley expressions or expressions that change abruptly without natural transition.",
+      "gender": "",
+      "age": "",
+      "skin_tone_and_texture": "Waxy, plastic-looking skin with visible artifacts and inconsistent texture resolution across the frame.",
+      "facial_features": "Asymmetric facial features, extra fingers or limbs, teeth that appear blurry or malformed.",
+      "number_of_subjects": 0,
+      "number_of_arms": 0,
+      "number_of_legs": 0
+    }
+  ],
+  "background_setting": "A poorly rendered, flat background with visible seams, repeated textures, and inconsistent depth cues. The environment lacks volumetric depth and appears as a painted backdrop rather than a three-dimensional space. Vegetation looks like flat cutouts with no volumetric depth. The background appears to have been composited from multiple source materials at different resolutions, creating visible seams and edge artifacts where elements meet. Textures swim and shift across surfaces in a way that breaks the illusion of solidity \u2014 patterns drift laterally rather than staying anchored to the geometry they belong to. Background elements flicker in and out of existence between frames, particularly at the edges of the field of view. The rendering resolution is visibly lower for distant elements, creating a jarring transition between near and far objects. Cloud textures repeat obviously in the sky with visible tiling. Water surfaces lack proper reflection and refraction, appearing as flat animated textures. Fog and atmospheric effects pop in and out rather than smoothly transitioning. Trees and vegetation exhibit obvious LOD (level-of-detail) switching. Building facades have inconsistent window spacing and pattern repetition. The overall scene feels like a poorly assembled collage of individually rendered elements rather than a coherent whole.",
+  "lighting": {
+    "conditions": "Harsh, flat lighting with no natural variation. The scene appears uniformly lit as if by a single overhead fluorescent light, removing all sense of depth and atmosphere.",
+    "direction": "Inconsistent light sources \u2014 shadows point in multiple contradictory directions, breaking physical plausibility.",
+    "shadows": "Hard-edged, unrealistic shadows that pop in and out of existence between frames. Some objects cast no shadows while others have impossibly dark ones that don't animate smoothly with the object's motion. Shadow edges exhibit visible staircase aliasing artifacts. Shadow maps appear to have been rendered at extremely low resolution, creating blocky patterns. Self-shadowing on characters shows visible peter-panning artifacts where shadows detach from their source. Contact shadows between objects and the ground appear and disappear as objects move slightly. Shadow color is pure black with no ambient contribution, creating an unnaturally harsh contrast that flattens the image. Multiple shadow cascades have visible boundaries where resolution changes. The shadow rendering appears to be temporally unstable \u2014 even static objects have shadows that shimmer and crawl frame to frame, breaking the illusion of a stable light source.",
+    "illumination_effect": "No bounce light, no ambient occlusion, no subtle color interactions between surfaces. The scene looks like a poorly lit 3D render from the early 2000s."
+  },
+  "aesthetics": {
+    "composition": "Cluttered, poorly framed composition with no clear focal point. Important elements are cut off by the frame edges. The rule of thirds is completely ignored, leading to an unbalanced and visually unpleasant arrangement.",
+    "color_scheme": "Oversaturated, garish colors that clash violently. Color banding is visible in gradient areas. The overall palette feels artificial and digitally processed rather than natural.",
+    "mood_atmosphere": "Unsettling, uncanny atmosphere that fails to evoke any intended emotional response. The scene feels lifeless and sterile despite attempting to portray dynamic action.",
+    "patterns": "Visible tiling artifacts in textures, moir\u00e9 patterns, and aliasing on edges."
+  },
+  "cinematography": {
+    "camera_motion": "Extremely shaky, unstable camera with visible rolling shutter artifacts. The motion is jerky and discontinuous, causing motion sickness and making the scene impossible to follow.",
+    "framing": "Poorly framed shots that cut off important elements and include unnecessary empty space.",
+    "camera_angle": "Awkward, disorienting camera angles that provide no useful spatial information about the scene. The camera path exhibits visible mathematical artifacts suggesting simple interpolation between keyframes rather than natural camera operation. Camera motion is completely disconnected from the scene content \u2014 panning away from action, dollying during dialogue, and shaking during still moments. The camera appears to pass through solid objects occasionally. Zoom is applied digitally rather than optically, revealing progressively worse resolution. Camera motion exhibits non-physical acceleration profiles \u2014 instant starts and stops rather than smooth ease-in/ease-out. Rolling shutter simulation is applied inconsistently, present in some frames but not others. The camera occasionally exhibits impossible motion like teleporting between positions. Virtual camera stabilization creates an uncanny floating sensation disconnected from any physical camera rig.",
+    "depth_of_field": "Uniform focus throughout, creating a flat, documentary-like appearance with no cinematic depth separation.",
+    "focus": "Soft, out-of-focus imagery with visible chromatic aberration and lens distortion that was not corrected in post-processing.",
+    "lens_focal_length": "Inappropriate focal length causing barrel distortion and unnatural perspective compression."
+  },
+  "style_medium": "Low quality compressed digital video with visible encoding artifacts",
+  "artistic_style": "Amateur, unpolished with inconsistent visual style",
+  "context": "A poorly produced video with numerous technical and artistic flaws that detract from any intended narrative or visual impact.",
+  "actions": [
+    {
+      "time": "0:00-0:08",
+      "description": "Subjects attempt to move but their motion is jerky, temporally inconsistent, and physically implausible. Background elements flicker and shift between frames."
+    }
+  ],
+  "text_and_signage_elements": [],
+  "segments": [
+    {
+      "segment_index": 0,
+      "time_range": "0:00-0:08",
+      "description": "A single continuous shot suffering from severe temporal inconsistencies \u2014 subjects that morph and deform between frames, backgrounds that shift and wobble, and rendering quality that fluctuates visibly over time. Motion blur is applied incorrectly, smearing in directions that don't match actual movement. Frame-to-frame coherence breaks down with individual pixels changing color randomly in flat areas. Texture detail level fluctuates between frames as if the rendering budget varied shot to shot. Color grading drifts over the duration with no creative motivation. Noise patterns change between frames in ways that draw attention rather than being invisible. Overall visual quality degrades progressively from start to finish.",
+      "key_changes": "No meaningful progression or narrative development. Visual quality degrades over time.",
+      "camera": "Unstable, poorly controlled camera work with visible mathematical interpolation artifacts."
+    }
+  ],
+  "transitions": [],
+  "temporal_caption": "The scene opens at 0.0 seconds with a poorly rendered establishing shot that immediately reveals low production quality. At 1.0 seconds, subjects begin to move but their motion is jerky and inconsistent, with limbs bending at unnatural angles and objects clipping through each other. From 2.0 to 4.0 seconds, the camera shakes violently while the scene exhibits visible compression artifacts, color banding in the sky, and flickering in the shadows. Between 4.0 and 6.0 seconds, temporal coherence breaks down as elements appear and disappear between frames, textures swim and morph unnaturally, and the lighting shifts abruptly without physical cause. In the final 2 seconds, the overall visual quality deteriorates further with increasing noise, blur, and a general loss of spatial coherence that makes the scene nearly unwatchable. Additionally, the frame rate appears inconsistent with visible judder and stuttering throughout. Color temperature shifts randomly between warm and cool tones with no motivation. The encode quality degrades in complex regions showing macro-blocking and mosquito noise around moving edges. Temporal noise patterns are spatially correlated, creating swimming artifacts on flat surfaces.",
+  "audio_description": "",
+  "physical_realism": "No adherence to physical laws. Objects defy gravity, pass through solid surfaces, and change mass and momentum without cause. Fluid dynamics, cloth simulation, and rigid body physics are all fundamentally broken. Furthermore, conservation of energy is violated as objects gain or lose kinetic energy spontaneously. Elastic collisions produce inelastic results and vice versa. Surface friction is inconsistent \u2014 objects slide on rough surfaces while sticking to smooth ones. Air resistance appears to affect only some objects while others move through the atmosphere unimpeded."
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+    "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' }}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- for message in messages %}\n    {%- if message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|vision_start|><|image_pad|><|vision_end|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|vision_start|><|video_pad|><|vision_end|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content_item in message.content %}\n                {%- if 'text' in content_item %}\n                    {{- content_item.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and message.content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|vision_start|><|image_pad|><|vision_end|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|vision_start|><|video_pad|><|vision_end|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n"
+}

checkpoint.json

@@ -0,0 +1 @@
+{}

config.json

@@ -0,0 +1,260 @@
+{
+  "allow_patterns_overrides": [
+    "*/*.safetensors"
+  ],
+  "architectures": [
+    "Cosmos3ForConditionalGeneration"
+  ],
+  "image_token_id": 151655,
+  "model": {
+    "_recursive_": false,
+    "_target": "omni_mot_model",
+    "config": {
+      "_type": "omni_mot_model_config",
+      "action_gen": true,
+      "activation_checkpointing": {
+        "_type": "activation_checkpointing_config",
+        "determinism_check": "default",
+        "mode": "full",
+        "preserve_rng_state": true,
+        "save_ops_regex": [
+          "fmha"
+        ]
+      },
+      "causal_training_strategy": "none",
+      "diffusion_expert_config": {
+        "_type": "diffusion_expert_config",
+        "base_fps": 24,
+        "enable_fps_modulation": true,
+        "load_weights_from_pretrained": false,
+        "max_vae_latent_side_after_patchify": 20,
+        "patch_spatial": 2,
+        "position_embedding_type": "unified_3d_mrope",
+        "rope_h_extrapolation_ratio": 1.0,
+        "rope_t_extrapolation_ratio": 1.0,
+        "rope_w_extrapolation_ratio": 1.0,
+        "timestep_range": 1.0,
+        "unified_3d_mrope_reset_spatial_ids": true,
+        "unified_3d_mrope_temporal_modality_margin": 15000
+      },
+      "ema": {
+        "_type": "ema_config",
+        "enabled": false,
+        "iteration_shift": 0,
+        "rate": 0.1
+      },
+      "fixed_step_sampler_config": null,
+      "input_caption_key": "ai_caption",
+      "input_image_key": "images",
+      "input_video_key": "video",
+      "joint_attn_implementation": "two_way",
+      "latent_downsample_factor": 16,
+      "lbl": {
+        "_type": "lbl_config",
+        "coeff_gen": null,
+        "coeff_und": null,
+        "method": "local"
+      },
+      "log_enc_time_every_n": 100,
+      "lora_alpha": 32,
+      "lora_enabled": false,
+      "lora_rank": 16,
+      "lora_target_modules": "q_proj_moe_gen,k_proj_moe_gen,v_proj_moe_gen,o_proj_moe_gen",
+      "max_action_dim": 64,
+      "max_num_tokens_after_packing": 74000,
+      "natten_parameter_list": null,
+      "net": null,
+      "num_embodiment_domains": 32,
+      "parallelism": {
+        "_type": "parallelism_config",
+        "cfg_parallel_shard_degree": 1,
+        "compile_dynamic": true,
+        "compiled_region": "language",
+        "context_parallel_shard_degree": 1,
+        "coordinate_descent_tuning": false,
+        "data_parallel_replicate_degree": 1,
+        "data_parallel_shard_degree": 128,
+        "enable_inference_mode": false,
+        "max_autotune_pointwise": false,
+        "precision": "bfloat16",
+        "use_cuda_graphs": false,
+        "use_torch_compile": true
+      },
+      "rectified_flow_inference_config": {
+        "_type": "rectified_flow_inference_config",
+        "num_train_timesteps": 1000,
+        "scheduler_type": "unipc",
+        "shift": 1,
+        "use_dynamic_shifting": false
+      },
+      "rectified_flow_training_config": {
+        "_type": "rectified_flow_training_config",
+        "action_loss_weight": 10.0,
+        "high_sigma_ratio": 0.05,
+        "high_sigma_timesteps_max": 1000,
+        "high_sigma_timesteps_min": 995,
+        "image_loss_scale": null,
+        "independent_action_schedule": false,
+        "independent_sound_schedule": false,
+        "loss_scale": 10.0,
+        "normalize_loss_by_active": false,
+        "shift": {
+          "256": 3,
+          "480": 5,
+          "720": 10
+        },
+        "shift_action": null,
+        "shift_sound": null,
+        "sound_loss_scale": 2.0,
+        "train_time_action_distribution": "logitnormal",
+        "train_time_image_distribution": "logitnormal",
+        "train_time_sound_distribution": "logitnormal",
+        "train_time_video_distribution": "waver",
+        "train_time_weight": "uniform",
+        "use_discrete_rf": false,
+        "use_dynamic_shift": false,
+        "use_high_sigma_strategy": false,
+        "use_high_sigma_strategy_action": false,
+        "use_high_sigma_strategy_sound": false
+      },
+      "resolution": "720",
+      "sound_dim": 64,
+      "sound_gen": true,
+      "sound_latent_fps": 25,
+      "sound_tokenizer": {
+        "_target": "avae_interface",
+        "audio_channels": 2,
+        "avae_config_path": "",
+        "avae_path": "pretrained/tokenizers/audio/avae/avae_48k_noncausal_25hz_64ch.ckpt",
+        "bucket_name": "bucket",
+        "hop_size": 1920,
+        "io_channels": 64,
+        "latent_mean": null,
+        "latent_std": null,
+        "normalization_type": "none",
+        "normalize_latents": false,
+        "object_store_credential_path_pretrained": "credentials/gcp_training.secret",
+        "sample_rate": 48000,
+        "tanh_clamp": 0.995,
+        "tanh_input_scale": 1.5,
+        "tanh_output_scale": 3.5
+      },
+      "state_ch": 48,
+      "state_t": 300,
+      "tokenizer": {
+        "_target": "wan2pt2_vae_interface",
+        "bucket_name": "bucket",
+        "chunk_duration": 93,
+        "encode_bucket_multiple": null,
+        "encode_chunk_frames": {
+          "256": 68,
+          "480": 24,
+          "720": 12
+        },
+        "encode_exact_durations": [
+          17,
+          61,
+          73
+        ],
+        "keep_decoder_cache": false,
+        "object_store_credential_path_pretrained": "credentials/gcp_training.secret",
+        "spatial_compression_factor": 16,
+        "temporal_compression_factor": 4,
+        "temporal_window": null,
+        "use_streaming_encode": false,
+        "vae_path": "pretrained/tokenizers/video/wan2pt2/Wan2.2_VAE.pth"
+      },
+      "video_temporal_causal": false,
+      "vision_gen": true,
+      "vlm_config": {
+        "_type": "vlm_config",
+        "layer_module": null,
+        "model_instance": {
+          "_target": "qwen3_vl_text_for_causal_lm",
+          "config": {
+            "_target": "create_vlm_config",
+            "base_config": {
+              "_target": "qwen3_vl_mot_config_from_json_file",
+              "json_file": "cosmos3://vfm/models/vlm/qwen3_vl/configs/Qwen3-VL-32B-Instruct.json"
+            },
+            "qk_norm_for_text": true
+          }
+        },
+        "model_name": "nvidia/Cosmos3-Super-Reasoner",
+        "pretrained_weights": {
+          "_type": "pretrained_weights_config",
+          "backbone_path": "s3://bucket/cosmos3/pretrained/huggingface/Cosmos-Reason/Cosmos3-Super-Reasoner-b6df0d1/",
+          "checkpoint_format": null,
+          "credentials_path": "credentials/gcp_checkpoint.secret",
+          "enable_gcs_patch_in_boto3": true,
+          "enabled": false
+        },
+        "qk_norm": false,
+        "tie_word_embeddings": false,
+        "tokenizer": {
+          "_target": "create_qwen2_tokenizer_with_download",
+          "config_variant": "gcp",
+          "pretrained_model_name": "Qwen/Qwen3-VL-32B-Instruct"
+        },
+        "use_system_prompt": false
+      }
+    }
+  },
+  "model_type": "cosmos3_omni",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "dtype": "bfloat16",
+    "eos_token_id": 151645,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 5120,
+    "initializer_range": 0.02,
+    "intermediate_size": 25600,
+    "max_position_embeddings": 262144,
+    "model_type": "qwen3_vl_text",
+    "num_attention_heads": 64,
+    "num_hidden_layers": 64,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        24,
+        20,
+        20
+      ],
+      "rope_type": "default"
+    },
+    "rope_theta": 5000000,
+    "use_cache": true,
+    "vocab_size": 151936
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.0.dev0",
+  "video_token_id": 151656,
+  "vision_config": {
+    "deepstack_visual_indexes": [
+      8,
+      16,
+      24
+    ],
+    "depth": 27,
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "qwen3_vl",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 5120,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652
+}
+

generation_config.json

@@ -0,0 +1,14 @@
+{
+    "bos_token_id": 151643,
+    "pad_token_id": 151643,
+    "do_sample": true,
+    "eos_token_id": [
+        151645,
+        151643
+    ],
+    "top_p": 0.8,
+    "top_k": 20,
+    "temperature": 0.7,
+    "repetition_penalty": 1.0,
+    "transformers_version": "4.56.0"
+}

load_cosmos3_modelopt.py

@@ -0,0 +1,256 @@
+#!/usr/bin/env python
+"""
+Drop-in loader for the weight-only NVFP4 / FP8 (NVIDIA ModelOpt) Cosmos3-Super checkpoint
+saved in the ROUND-TRIPPABLE format (modelopt_state.pth present -- see repackage_for_hf.py).
+
+Current diffusers + accelerate + modelopt treat this path as experimental; each shim below
+works around a specific, source-verified version-skew gap. None modify the checkpoint:
+
+  1. enable_huggingface_checkpointing()
+     Registers ModelOpt's HF handlers so `from_pretrained` restores the quantized module
+     structure from modelopt_state.pth before weights load.
+
+  2. set_module_tensor_to_device patch (parameter materialization)
+     The modelopt_state restore runs inside diffusers' meta-device init, so each quantized
+     weight is a QTensorWrapper whose storage is a META tensor carrying dequant metadata.
+     `param.data = real` can't cross meta<->real, and accelerate's rebuild both rejects
+     requires_grad and discards metadata. We REPLACE the parameter with a fresh wrapper
+     around the loaded bytes + the existing metadata. Two extra duties here:
+       - payload dtype restore: diffusers casts floating params to torch_dtype during load
+         when no hf_quantizer is present (model_loading_utils.py -- their own TODO flags
+         float8). FP8 payloads are floating and arrive cast to bf16; we cast back to the
+         wrapper's payload dtype (exact: every e4m3fn value round-trips through bf16).
+         NVFP4's uint8 payload is never cast, so this is a no-op there.
+       - direct-to-GPU materialization: staging payloads on CPU would put the whole packed
+         model in system RAM (FP8: ~64 GB on a 32 GB box -> OOM-killed; NVFP4 only survived
+         because uncast tensors stay mmap-backed). Wrappers go straight to `materialize_device`.
+
+  3. Post-restore quantizer re-disable
+     modelopt_state replays the QUANT CONFIG, not imperative `.disable()` calls made after
+     quantize. The NVFP4 build used NVFP4_DEFAULT_CFG (activation quantization ON in-config)
+     with activations disabled imperatively -- so the restored model comes back with ~1806
+     dynamic fake-quant activation quantizers active: ~10x slower, fatter, and quantizing
+     activations the validated regime never quantized. We re-apply weight-only + spare
+     disabling after load. (FP8's config had the disables baked in; this is then a no-op.)
+
+  4. Full bf16 normalization (the validated serve regime)
+     Cosmos3OmniTransformer pins time_embedder to fp32 via _keep_in_fp32_modules
+     (transformer_cosmos3.py:297); the quantized model runs all-bf16. Cast floating buffers
+     and non-wrapper floating params to bf16, retarget wrapper dequant dtype to bf16, and
+     register bf16 input-cast pre-hooks on the time embedders.
+
+  5. NVIDIAModelOptQuantizer.create_quantized_param patch
+     Only relevant if a checkpoint carries an embedded quantization_config (hf_quantizer
+     path): requires_grad only for float/complex tensors. Kept for robustness.
+
+Usage (library):
+    from load_cosmos3_modelopt import load_pipe
+    pipe = load_pipe("YOUR_HF_USERNAME/Cosmos3-Super-nvfp4")   # or a local dir
+    # NOTE: a bare pipe(prompt) call renders the pipeline DEFAULT: a 189-frame 720x1280
+    # video (~8 s at 24 fps) -- not a still. For a single-image smoke test, be explicit:
+    r = pipe("a red cube on a table", height=1024, width=1024, num_frames=1,
+             num_inference_steps=50, guidance_scale=4.0)
+    r.video[0].save("out.png")          # .video is the list of PIL frames; [0] is the image
+
+Usage (interactive -- lands in a REPL with `pipe` loaded):
+    CUDA_VISIBLE_DEVICES=0 python -i load_cosmos3_modelopt.py ./cosmos3-super-nvfp4-hf
+"""
+import os
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch
+
+SPARE_SUBSTRINGS = [
+    "time_embedder", "proj_in", "proj_out", "lm_head", "embed", "norm", "audio_proj",
+]
+
+
+def _qtensor_wrapper_cls():
+    try:
+        from modelopt.torch.quantization.qtensor.base_qtensor import QTensorWrapper
+        return QTensorWrapper
+    except Exception:
+        return None
+
+
+def _patch_modelopt_quantizer() -> None:
+    """diffusers pre-quantized hf_quantizer branch: only float/complex tensors may require grad."""
+    import diffusers.quantizers.modelopt.modelopt_quantizer as moq
+
+    if getattr(moq.NVIDIAModelOptQuantizer, "_cqp_patched", False):
+        return
+    _orig = moq.NVIDIAModelOptQuantizer.create_quantized_param
+
+    def _cqp(self, model, param_value, param_name, target_device, *args, **kwargs):
+        if self.pre_quantized:
+            module, tname = moq.get_module_from_name(model, param_name)
+            needs_grad = param_value.is_floating_point() or param_value.is_complex()
+            module._parameters[tname] = torch.nn.Parameter(
+                param_value.to(device=target_device), requires_grad=needs_grad
+            )
+            return
+        return _orig(self, model, param_value, param_name, target_device, *args, **kwargs)
+
+    moq.NVIDIAModelOptQuantizer.create_quantized_param = _cqp
+    moq.NVIDIAModelOptQuantizer._cqp_patched = True
+
+
+def _patch_qtensor_loading() -> None:
+    """Materialize restored (meta) QTensorWrapper params: replace the parameter object with
+    a fresh wrapper around the loaded bytes, restoring the payload dtype (undoes diffusers'
+    float cast on FP8) and landing directly on the target device (keeps payloads out of RAM)."""
+    import diffusers.models.model_loading_utils as mlu
+
+    if getattr(mlu, "_qtw_inplace_patched", False):
+        return
+    QTensorWrapper = _qtensor_wrapper_cls()
+    if QTensorWrapper is None:
+        return  # different modelopt layout; nothing to patch
+
+    _orig = mlu.set_module_tensor_to_device
+    stats = {"materialized": 0, "target_device": None}
+
+    def _patched(model, tensor_name, device, value=None, *args, **kwargs):
+        if value is not None:
+            module, leaf = model, tensor_name
+            if "." in tensor_name:
+                mod_path, leaf = tensor_name.rsplit(".", 1)
+                try:
+                    module = model.get_submodule(mod_path)
+                except AttributeError:
+                    module = None
+            if module is not None:
+                cur = getattr(module, "_parameters", {}).get(leaf)
+                if isinstance(cur, QTensorWrapper):
+                    tgt = stats.get("target_device") or device
+                    module._parameters[leaf] = QTensorWrapper(
+                        value.to(device=tgt, dtype=cur.data.dtype),  # exact cast-back for fp8
+                        metadata=dict(cur.metadata),
+                    )
+                    stats["materialized"] += 1
+                    return None
+        return _orig(model, tensor_name, device, value=value, *args, **kwargs)
+
+    mlu.set_module_tensor_to_device = _patched
+    mlu._qtw_inplace_patched = True
+    mlu._qtw_stats = stats
+
+
+def _enforce_weight_only(transformer) -> None:
+    """Re-apply the validated weight-only + spare regime: the state replay re-enables any
+    quantizers that were disabled imperatively after quantize (NVFP4 default cfg has
+    activation quantization ON in-config)."""
+    n_act = n_spare = 0
+    for name, m in transformer.named_modules():
+        if not (name.endswith("_quantizer") and hasattr(m, "disable")):
+            continue
+        if name.endswith("weight_quantizer"):
+            if any(s in name for s in SPARE_SUBSTRINGS):
+                if getattr(m, "is_enabled", False):
+                    n_spare += 1
+                m.disable()
+        else:
+            if getattr(m, "is_enabled", False):
+                n_act += 1
+            m.disable()
+    print(f"[load] re-disabled quantizers the state replay re-enabled: "
+          f"{n_act} activation, {n_spare} spare-weight")
+
+
+def _apply_dtype_nudges(transformer) -> None:
+    """Reproduce the validated all-bf16 serve regime (all no-ops where already aligned)."""
+    QTensorWrapper = _qtensor_wrapper_cls() or ()
+
+    n_buf = n_par = n_meta = 0
+    for m in transformer.modules():
+        for bn, buf in list(m._buffers.items()):
+            if buf is not None and buf.is_floating_point() and buf.dtype != torch.bfloat16:
+                m._buffers[bn] = buf.to(torch.bfloat16)
+                n_buf += 1
+    for _, p in transformer.named_parameters():
+        if isinstance(p, QTensorWrapper):
+            d = p.metadata.get("dtype")
+            if isinstance(d, torch.dtype) and d.is_floating_point and d != torch.bfloat16:
+                p.metadata["dtype"] = torch.bfloat16   # dequant target only; payload untouched
+                n_meta += 1
+            continue  # packed payloads: never cast
+        if p.is_floating_point() and p.dtype != torch.bfloat16:
+            p.data = p.data.to(torch.bfloat16)
+            n_par += 1
+    print(f"[load] normalized to bf16: {n_par} params, {n_buf} buffers; "
+          f"retargeted {n_meta} dequant dtypes")
+
+    def _cast_bf16(_m, args):
+        return tuple(
+            a.to(torch.bfloat16)
+            if torch.is_tensor(a) and a.is_floating_point() and a.dtype != torch.bfloat16
+            else a
+            for a in args
+        )
+
+    for name, m in transformer.named_modules():
+        if "time_embedder" in name and hasattr(m, "linear_1"):
+            m.register_forward_pre_hook(_cast_bf16)
+
+
+def load_pipe(
+    model_id_or_path: str,
+    *,
+    torch_dtype=torch.bfloat16,
+    enable_safety_checker: bool = False,
+    device: str = "cuda",
+    materialize_device: str | None = "cuda",   # packed weights stream straight here (RAM stays low)
+    **kwargs,
+):
+    """Load a ModelOpt-quantized Cosmos3-Super pipeline with all load-time fixes applied."""
+    from diffusers import Cosmos3OmniPipeline
+    from modelopt.torch.opt import enable_huggingface_checkpointing
+
+    enable_huggingface_checkpointing()   # must run before from_pretrained
+    _patch_modelopt_quantizer()
+    _patch_qtensor_loading()
+
+    import diffusers.models.model_loading_utils as mlu
+    if hasattr(mlu, "_qtw_stats"):
+        mlu._qtw_stats["materialized"] = 0
+        mlu._qtw_stats["target_device"] = materialize_device
+
+    pipe = Cosmos3OmniPipeline.from_pretrained(
+        model_id_or_path,
+        torch_dtype=torch_dtype,
+        enable_safety_checker=enable_safety_checker,
+        **kwargs,
+    )
+
+    n_mat = getattr(mlu, "_qtw_stats", {}).get("materialized", 0)
+    print(f"[load] materialized {n_mat} packed quantized weight tensors")
+
+    transformer = getattr(pipe, "transformer", None)
+    if transformer is not None:
+        _enforce_weight_only(transformer)
+        _apply_dtype_nudges(transformer)
+
+    pipe = pipe.to(device)
+
+    QTensorWrapper = _qtensor_wrapper_cls()
+    if QTensorWrapper is not None and transformer is not None:
+        n_live = sum(1 for p in transformer.parameters() if isinstance(p, QTensorWrapper))
+        print(f"[load] {n_live} quantized weight wrappers active after move to {device}")
+        if n_mat and not n_live:
+            print("[load] WARNING: wrappers were lost during .to() -- do not render; report this")
+
+    return pipe
+
+
+if __name__ == "__main__":
+    import sys
+
+    path = sys.argv[1] if len(sys.argv) > 1 else "./cosmos3-super-nvfp4-hf"
+    print(f"[load] loading {path} ...")
+    pipe = load_pipe(path)
+    print("[load] OK -- `pipe` is ready.")
+    print("       NOTE: bare pipe(prompt) renders a 189-frame 720x1280 video (pipeline default).")
+    print("       single-still smoke test:")
+    print("       r = pipe('a red cube on a table', height=1024, width=1024, num_frames=1,")
+    print("                num_inference_steps=50, guidance_scale=4.0); r.video[0].save('out.png')")

model_index.json

@@ -0,0 +1,28 @@
+{
+  "_class_name": "Cosmos3OmniDiffusersPipeline",
+  "_diffusers_version": "0.37.1",
+  "scheduler": [
+    "diffusers",
+    "UniPCMultistepScheduler"
+  ],
+  "text_tokenizer": [
+    "transformers",
+    "Qwen2TokenizerFast"
+  ],
+  "transformer": [
+    "diffusers",
+    "Cosmos3OmniTransformer"
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKLWan"
+  ],
+  "vision_encoder": [
+    "transformers",
+    "Qwen3VLVisionModel"
+  ],
+  "sound_tokenizer": [
+    "diffusers",
+    "Cosmos3AVAEAudioTokenizer"
+  ]
+}

preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 16777216,
+        "shortest_edge": 65536
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "image_processor_type": "Qwen2VLImageProcessorFast"
+}

quantize_cosmos3_super_streaming.py

@@ -0,0 +1,401 @@
+#!/usr/bin/env python
+"""
+Stream-quantize NVIDIA Cosmos3-Super's transformer to weight-only FP8 or NVFP4,
+WITHOUT ever materializing the full ~128 GB BF16 model.
+
+WHY THIS EXISTS
+---------------
+The diffusers <-> ModelOpt quantize-on-load path materializes the full BF16 model
+on the GPU before compressing, which needs ~128 GB of VRAM for a 64B model and
+won't spill to CPU. That can't fit a 96 GB card. This script instead replicates
+ModelOpt's own `init_quantized_weights` (the engine behind `--low_memory_mode` in
+their llm_ptq example), which is the tool NVIDIA built for exactly "I can RUN the
+FP8 model but can't QUANTIZE it naively":
+
+    1. Build the transformer EMPTY on the meta device (zero real memory).
+    2. mtq.quantize(...)  -> insert quantizers (weight-only; no model execution).
+    3. mtq.compress(...)  -> set up REAL compressed (FP8/NVFP4) parameter shapes.
+    4. infer a device_map from those *compressed* sizes.
+    5. load_checkpoint_and_dispatch(...) -> stream the BF16 shards straight into
+       compressed form, one shard at a time. Per-tensor weight scales are computed
+       from each weight as it lands. The full BF16 is NEVER resident.
+
+Peak memory ~= compressed size + one shard:
+    FP8   ~65 GB  -> fits the 96 GB RTX 6000 Pro alone, comfortably.
+    NVFP4 ~36 GB  -> fits with enormous margin.
+
+Both formats are WEIGHT-ONLY, so neither runs a calibration forward pass -- the
+64B model never executes. The only difference between them is the quant config;
+ModelOpt does the FP8 per-tensor scaling and the NVFP4 4-bit block-scale packing
+internally, so you get a fair FP8-vs-NVFP4 comparison from one script.
+
+Output is the ModelOpt "unified HF checkpoint" (safetensors + hf_quant_config.json),
+loadable by vLLM / TensorRT-LLM / diffusers.
+
+USAGE
+-----
+    python quantize_cosmos3_super_streaming.py --format fp8
+    python quantize_cosmos3_super_streaming.py --format nvfp4
+    # add --smoke to attempt a tiny render from the exported checkpoint afterward.
+
+Outputs go to ./cosmos3-super-<fmt>/ (override with --export-dir).
+"""
+
+import argparse
+import os
+
+# Reduce allocator fragmentation on the big card (cheap, always-on).
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch
+from accelerate import init_empty_weights, load_checkpoint_in_model
+from accelerate.utils import get_max_memory, infer_auto_device_map
+from accelerate.utils.dataclasses import CustomDtype
+from huggingface_hub import snapshot_download
+
+import modelopt.torch.quantization as mtq
+from modelopt.torch.export import export_hf_checkpoint
+
+# Cosmos3 classes require diffusers built from git main (already installed in the venv).
+from diffusers import Cosmos3OmniTransformer
+
+SRC_REPO = "nvidia/Cosmos3-Super"
+
+# ---------------------------------------------------------------------------
+# Layers to KEEP IN BF16 (never quantize). Matched as substrings of module names.
+# Mirrors the Cosmos3-Nano-FP8 community recipe + what --inspect showed for Super:
+# keep embeddings, norms, the bundled Qwen3 reasoner head, time/modality adapters,
+# and the in/out projections. These are tiny relative to the 64 transformer blocks
+# but quality-sensitive, so they cost almost nothing to leave in BF16.
+#
+# NOTE: substrings are deliberately specific. "proj_in"/"proj_out" do NOT match the
+# attention "add_q_proj"/"to_out"/"down_proj" etc. (different names), so the 64 MMDiT
+# blocks' attention + MLP linears still get quantized.
+# ---------------------------------------------------------------------------
+SPARE_SUBSTRINGS = [
+    "time_embedder",
+    "proj_in",
+    "proj_out",
+    "lm_head",
+    "embed",        # token / position embeddings
+    "norm",         # layernorms / rmsnorms
+    "audio_proj",   # audio modality adapter
+]
+
+
+def _is_spare(module_name: str) -> bool:
+    return any(s in module_name for s in SPARE_SUBSTRINGS)
+
+
+def build_quant_cfg(fmt: str) -> dict:
+    """Return a WEIGHT-ONLY quant config for the chosen format.
+
+    We start from ModelOpt's weight-only presets so the heavy lifting (the FP8
+    per-tensor scale and the NVFP4 E2M1 + block-scale packing) is ModelOpt's code,
+    not ours. We only ensure activations stay off (weight-only) and normalize to a
+    dict so the spare-exclusion below is unambiguous.
+    """
+    if fmt == "fp8":
+        # Per-tensor E4M3 weights, everything else in BF16. This is the exact dict
+        # that previously passed "Inserted 2709 quantizers" under mtq.
+        return {
+            "quant_cfg": {
+                "*weight_quantizer": {"num_bits": (4, 3), "axis": None, "enable": True},
+                "*input_quantizer": {"enable": False},
+                "*output_quantizer": {"enable": False},
+                "*softmax_quantizer": {"enable": False},
+            },
+            "algorithm": "max",
+        }
+    elif fmt == "nvfp4":
+        # NVFP4: E2M1 4-bit weights, block_size 16, FP8 (E4M3) block scales + a FP32
+        # per-tensor scale. We base this on whichever NVFP4 config your installed
+        # modelopt actually ships: newer versions have a weight-only W4A16_NVFP4_CFG,
+        # older ones only NVFP4_DEFAULT_CFG (weights + activations).
+        #
+        # IMPORTANT: the weight-only disables must be baked INTO THE CONFIG, not just
+        # applied imperatively afterwards. modelopt_state (written by save_pretrained
+        # for the drop-in HF repo) replays the CONFIG on restore -- imperative
+        # .disable() calls made after quantize are not captured, so a checkpoint built
+        # from a bare NVFP4_DEFAULT_CFG restores with ~1806 dynamic activation
+        # quantizers active (~10x slower per step). enforce_weight_only_and_spare()
+        # below still runs as belt-and-braces for the live model.
+        import copy
+
+        base = getattr(mtq, "W4A16_NVFP4_CFG", None) or mtq.NVFP4_DEFAULT_CFG
+        cfg = copy.deepcopy(base)
+        cfg.setdefault("quant_cfg", {})
+        cfg["quant_cfg"]["*input_quantizer"] = {"enable": False}
+        cfg["quant_cfg"]["*output_quantizer"] = {"enable": False}
+        cfg["quant_cfg"]["*softmax_quantizer"] = {"enable": False}
+        for s in SPARE_SUBSTRINGS:
+            cfg["quant_cfg"][f"*{s}*weight_quantizer"] = {"enable": False}
+        return cfg
+    else:
+        raise ValueError(f"Unknown format: {fmt!r}")
+
+
+def enforce_weight_only_and_spare(model) -> tuple[int, int]:
+    """Make the model strictly WEIGHT-ONLY and keep SPARE layers in BF16.
+
+    Runs after mtq.quantize (on the meta model) and before mtq.compress. Walks every
+    inserted quantizer -- they appear as leaf modules whose name ends in '_quantizer':
+      - any NON-weight quantizer (input/output/softmax/bmm activation) -> DISABLED,
+        so activations stay BF16 regardless of which base config we started from.
+      - a weight quantizer on a SPARE layer (embeddings/norms/head/adapters) -> DISABLED,
+        so those weights stay BF16.
+
+    This is config-form agnostic (works whether quant_cfg was a dict or a list), which
+    is why we can feed it either the FP8 dict or modelopt's NVFP4 preset unchanged.
+
+    Returns (spare_weight_quantizers_disabled, activation_quantizers_disabled).
+    """
+    n_spare = 0
+    n_act = 0
+    for name, module in model.named_modules():
+        if not (name.endswith("_quantizer") and hasattr(module, "disable")):
+            continue
+        if name.endswith("weight_quantizer"):
+            parent = name.rsplit(".", 1)[0]
+            if _is_spare(parent):
+                module.disable()
+                n_spare += 1
+        else:
+            module.disable()
+            n_act += 1
+    return n_spare, n_act
+
+
+def compressed_device_map(model, gpu_mem_fraction: float = 0.85) -> dict:
+    """Build a device_map sized for the COMPRESSED weights.
+
+    Adapted from ModelOpt's init_quantized_weights.get_model_device_map: tell
+    accelerate that each compressed weight is FP8 (8-bit) or INT4 (4-bit) so the
+    map reflects ~65 GB / ~36 GB, not the 128 GB BF16 footprint. The result will
+    place essentially everything on GPU 0 (it fits), but GPU 1 + CPU stay available
+    as spill if a future/larger model needs them.
+    """
+    max_memory = {k: v * gpu_mem_fraction for k, v in get_max_memory().items()}
+
+    # Treat the first transformer block's class as un-splittable so a single block
+    # isn't torn across devices (keeps attention math on one device).
+    no_split = set()
+    for name, module in model.named_modules():
+        if name.endswith((".layers.0", ".blocks.0", ".transformer_blocks.0")):
+            no_split.add(module.__class__.__name__)
+
+    special_dtypes = {}
+    for name, module in model.named_modules():
+        if (
+            hasattr(module, "weight")
+            and hasattr(module, "weight_quantizer")
+            and getattr(module.weight_quantizer, "is_enabled", True)
+            and not getattr(module.weight_quantizer, "fake_quant", True)
+        ):
+            nb = module.weight_quantizer.num_bits
+            if isinstance(nb, tuple):  # e.g. (4,3) for FP8, (2,1) for NVFP4
+                nb = nb[0] + nb[1] + 1
+            special_dtypes[name + ".weight"] = CustomDtype.FP8 if nb == 8 else CustomDtype.INT4
+
+    return infer_auto_device_map(
+        model,
+        max_memory=max_memory,
+        no_split_module_classes=list(no_split),
+        special_dtypes=special_dtypes,
+    )
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--format", choices=["fp8", "nvfp4"], required=True,
+                    help="Weight-only quantization format to produce.")
+    ap.add_argument("--export-dir", default=None,
+                    help="Output dir (default: ./cosmos3-super-<format>).")
+    ap.add_argument("--gpu-mem-fraction", type=float, default=0.85,
+                    help="Fraction of each GPU's memory accelerate may use for placement.")
+    ap.add_argument("--smoke", action="store_true",
+                    help="Validate by rendering one 1024x1024 image from the in-memory model "
+                         "(runs before export; output: cosmos3_super_<fmt>_validate.png).")
+    args = ap.parse_args()
+
+    export_dir = args.export_dir or f"./cosmos3-super-{args.format}"
+    os.makedirs(export_dir, exist_ok=True)
+
+    print(f"[1/6] Resolving local checkpoint for {SRC_REPO} (transformer only)...")
+    # Pull just the transformer subfolder's config + safetensors shards locally.
+    local_root = snapshot_download(SRC_REPO, allow_patterns=["transformer/*"])
+    transformer_dir = os.path.join(local_root, "transformer")
+    print(f"      transformer dir: {transformer_dir}")
+
+    print("[2/6] Building EMPTY transformer on meta device (params on meta, buffers real)...")
+    config = Cosmos3OmniTransformer.load_config(transformer_dir)
+    # include_buffers=False is load-bearing: parameters still go to meta (the ~128 GB
+    # we're avoiding), but buffers (rotary frequencies, masks, etc.) get REAL storage.
+    # If buffers were meta too, accelerate's final dispatch -> model.to(device) crashes
+    # with "Cannot copy out of meta tensor". mtq.quantize runs *after* this context, so
+    # the quantizer scale buffers are real as well -> nothing meta survives to dispatch.
+    with init_empty_weights(include_buffers=False):
+        model = Cosmos3OmniTransformer.from_config(config)
+
+    print(f"[3/6] Inserting quantizers ({args.format}) on the meta model...")
+    quant_cfg = build_quant_cfg(args.format)
+    mtq.quantize(model, quant_cfg)  # no forward_loop: weight-only needs no calibration
+    n_spare, n_act = enforce_weight_only_and_spare(model)
+    print(f"      weight-only: disabled {n_act} activation quantizers; "
+          f"kept {n_spare} projection layers in BF16 (plus embeddings/norms/head)")
+
+    print("[4/6] Setting up compressed parameter shapes (mtq.compress)...")
+    # quant_gemm=False matches ModelOpt's low-memory loader; export handles the rest.
+    try:
+        mtq.compress(model, config=mtq.CompressConfig(quant_gemm=False))
+    except (AttributeError, TypeError):
+        # Older modelopt: compress takes no CompressConfig.
+        mtq.compress(model)
+
+    print("[5/6] Streaming BF16 shards into compressed form (this is the long step)...")
+    device_map = compressed_device_map(model, args.gpu_mem_fraction)
+    # Load weights into the compressed (meta) model WITHOUT accelerate's dispatch step.
+    # The all-in-one load_checkpoint_and_dispatch finishes with model.to(device), which
+    # crashes on the leftover meta _amax scratch buffers of the ~1,800 DISABLED quantizers
+    # (never written by the load). We load here, materialize those residual meta tensors
+    # ourselves, then export. The real compressed weights + scales are filled by this call.
+    load_checkpoint_in_model(
+        model,
+        checkpoint=transformer_dir,
+        device_map=device_map,
+        dtype=torch.bfloat16,
+    )
+
+    # Materialize any residual meta buffers/params as zeros on GPU so nothing meta reaches
+    # export. These belong to disabled (unused) quantizers, so zeros are inert; the enabled
+    # weight quantizers' scales were already computed during the load above.
+    n_fixed = 0
+    for _, module in model.named_modules():
+        for bname, buf in list(module._buffers.items()):
+            if buf is not None and getattr(buf, "is_meta", False):
+                module._buffers[bname] = torch.zeros(buf.shape, dtype=buf.dtype, device="cuda")
+                n_fixed += 1
+        for pname, par in list(module._parameters.items()):
+            if par is not None and getattr(par, "is_meta", False):
+                module._parameters[pname] = torch.nn.Parameter(
+                    torch.zeros(par.shape, dtype=par.dtype, device="cuda"), requires_grad=False
+                )
+                n_fixed += 1
+    if n_fixed:
+        print(f"      materialized {n_fixed} residual meta tensors (disabled-quantizer scratch)")
+
+    # Footprint report (compressed weights + buffers actually resident)
+    n_bytes = sum(p.numel() * p.element_size() for p in model.parameters() if p.device.type != "meta")
+    n_bytes += sum(b.numel() * b.element_size() for b in model.buffers() if b.device.type != "meta")
+    print(f"      live footprint: {n_bytes / 1e9:.1f} GB")
+
+    # Validate BEFORE export: render from the pristine post-load model. export_hf_checkpoint
+    # may mutate quantizer state (e.g. the QKV amax fusion), so we eyeball the real weights
+    # first. render_from_memory frees its own GPU memory before export reuses the card.
+    if args.smoke:
+        render_from_memory(model, args.format)
+
+    print(f"[6/6] Exporting unified HF checkpoint to {export_dir} ...")
+    with torch.inference_mode():
+        export_hf_checkpoint(model, export_dir=export_dir)
+    print(f"DONE. Quantized {args.format.upper()} checkpoint written to {export_dir}")
+
+
+def render_from_memory(model, fmt: str):
+    """Validate the quantization by rendering one image from the IN-MEMORY model.
+
+    This renders directly from the transformer we just quantized -- no reload. That's the
+    correct torch validation path: diffusers' torch round-trip uses mto.save/mto.restore,
+    whereas what we export is the *deployment* unified checkpoint (for vLLM-Omni / TRT-LLM).
+    Because mtq.compress used quant_gemm=False, the QuantLinears dequantize on the fly, so
+    the rendered image reflects exactly the FP8/NVFP4 weight rounding -- which is what we
+    want to eyeball. Best-effort: any failure here is reported but does not block export.
+    """
+    import gc
+
+    print(f"\n[validate] Rendering a 1024x1024 image from the in-memory {fmt.upper()} model...")
+    try:
+        from diffusers import Cosmos3OmniPipeline
+        from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
+
+        # Consolidate everything onto GPU 0: the compressed weights are already there; this
+        # pulls the small from_config buffers (rotary, etc.) over too. Safe now that the
+        # materialize-meta step left nothing on meta.
+        model.to("cuda")
+
+        # Dtype consistency at render time. Our model mixes FP8 (compressed) weights with BF16
+        # spare weights, while diffusers computes the timestep sinusoidal fresh in FP32 each
+        # step. That FP32 tensor then meets the BF16 time-embedder weights ->
+        # "mat1 and mat2 must have the same dtype". NVIDIA's uniform-BF16 run never sees this;
+        # our mixed model needs two nudges (runtime-only -- the exported weights are untouched):
+        #   (a) cast any FP32 buffers (from the empty-init) to BF16, leaving FP8 weights alone;
+        #   (b) cast time-embedder inputs to BF16 at the FP32->BF16 boundary.
+        for _module in model.modules():
+            for _bn, _buf in list(_module._buffers.items()):
+                if _buf is not None and _buf.dtype == torch.float32:
+                    _module._buffers[_bn] = _buf.to(torch.bfloat16)
+
+        def _cast_inputs_bf16(_m, args):
+            return tuple(
+                a.to(torch.bfloat16)
+                if torch.is_tensor(a) and a.is_floating_point() and a.dtype != torch.bfloat16
+                else a
+                for a in args
+            )
+
+        n_hooks = 0
+        for _name, _module in model.named_modules():
+            if "time_embedder" in _name and hasattr(_module, "linear_1"):
+                _module.register_forward_pre_hook(_cast_inputs_bf16)
+                n_hooks += 1
+        print(f"[validate] dtype-safety: cast fp32 buffers to bf16, hooked {n_hooks} time-embedder(s)")
+
+        # Pass OUR quantized transformer in so the pipeline does NOT reload it from the hub;
+        # it only fetches the small components (VAE, scheduler, tokenizer). The reasoner tower
+        # lives inside the transformer, so nothing large is double-loaded -> fits the 96 GB card.
+        pipe = Cosmos3OmniPipeline.from_pretrained(
+            SRC_REPO,
+            transformer=model,
+            torch_dtype=torch.bfloat16,
+            enable_safety_checker=False,  # skip the guardrail model for a local check
+        )
+        pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=3.0)
+        for name, comp in pipe.components.items():
+            if name != "transformer" and isinstance(comp, torch.nn.Module):
+                comp.to("cuda")
+
+        prompt = (
+            "A medium shot of a modern robotics research laboratory with white walls and a gray "
+            "floor. A robotic arm with a metallic finish is mounted on a clean white workbench, "
+            "its gripper positioned above a row of small colored objects. A large monitor on the "
+            "wall behind displays a software interface, brightly lit by overhead lights."
+        )
+        with torch.inference_mode():
+            result = pipe(
+                prompt=prompt,
+                negative_prompt="",
+                num_frames=1,            # single frame -> still image
+                height=1024,
+                width=1024,
+                num_inference_steps=50,
+                guidance_scale=4.0,
+                generator=torch.Generator(device="cuda").manual_seed(1234),
+            )
+        out_path = f"cosmos3_super_{fmt}_validate.png"
+        result.video[0].save(out_path)
+        print(f"[validate] Wrote {out_path}. Eyeball it for coherence; compare fp8 vs nvfp4 "
+              f"(same prompt + seed 1234, so differences are purely the format).")
+
+        # Free the pipeline's extra components/activations before export reuses the GPU.
+        del pipe, result
+        gc.collect()
+        torch.cuda.empty_cache()
+    except Exception as e:
+        import traceback
+        print(f"[validate] Render failed ({type(e).__name__}: {e}).")
+        print("[validate] This does NOT affect the quantized weights; export still proceeds below.")
+        traceback.print_exc()
+
+
+if __name__ == "__main__":
+    main()

repackage_for_hf.py

@@ -0,0 +1,111 @@
+#!/usr/bin/env python
+"""
+Re-save the in-memory quantized Cosmos3-Super transformer in the ROUND-TRIPPABLE
+ModelOpt HF format and assemble a complete drop-in diffusers repo around it.
+
+WHY THIS EXISTS
+---------------
+`export_hf_checkpoint()` writes a *deployment* checkpoint (hf_quant_config.json +
+packed weights) meant for TRT-LLM / vLLM, which load the packed format with their
+own kernels. diffusers' `from_pretrained` does NOT reconstruct the quantized
+modules from that. Instead, on load, ModelOpt restores the quantized module
+structure from a file named `modelopt_state.pth` in the checkpoint dir (via
+`restore_from_modelopt_state`). That file is written by `save_pretrained()` once
+`enable_huggingface_checkpointing()` has been called -- and is NOT written by
+`export_hf_checkpoint()`. So for a diffusers drop-in repo, re-save with
+`save_pretrained`. This script does that and then copies the non-transformer
+pipeline components (VAE, tokenizers, model_index.json) from an existing assembled
+dir, producing a self-contained repo you can publish.
+
+Run in the ModelOpt venv, from the directory containing serve_cosmos3_diffusers.py:
+    CUDA_VISIBLE_DEVICES=0 python repackage_for_hf.py --format nvfp4 \
+        --serve-dir ./cosmos3-super-nvfp4-serve \
+        --out-dir   ./cosmos3-super-nvfp4-hf \
+        [--cache ./cosmos3-cache]        # optional: faster rebuild if you have it
+
+Then verify the result loads + renders:
+    python -i load_cosmos3_modelopt.py ./cosmos3-super-nvfp4-hf
+"""
+import argparse
+import json
+import os
+import pathlib
+import shutil
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch  # noqa: F401  (ensures CUDA init / dtype availability)
+from modelopt.torch.opt import enable_huggingface_checkpointing
+
+# Reuse the proven in-memory build (empty-on-meta -> quantize -> compress -> stream).
+from serve_cosmos3_diffusers import build_quantized_transformer, try_restore_quantized
+
+QUANT_TYPE = {"fp8": "FP8", "nvfp4": "NVFP4"}
+
+
+def ensure_loadable_config(transformer_dir: str, fmt: str) -> None:
+    """If save_pretrained wrote a quantization_config, make sure diffusers can construct
+    it: NVIDIAModelOptConfig needs `quant_type`, and a truthy `modelopt_config` avoids the
+    buggy get_config_from_quant_type() builder. (Structure restore itself comes from
+    modelopt_state.pth; this just keeps config parsing from crashing on load.)"""
+    cfg_path = pathlib.Path(transformer_dir) / "config.json"
+    cfg = json.loads(cfg_path.read_text())
+    qc = cfg.get("quantization_config")
+    if isinstance(qc, dict):
+        qc["quant_type"] = QUANT_TYPE[fmt]
+        qc.setdefault("weight_only", True)
+        if not qc.get("modelopt_config"):
+            qc["modelopt_config"] = {"quant_cfg": {}, "algorithm": "max"}
+        cfg["quantization_config"] = qc
+        cfg_path.write_text(json.dumps(cfg, indent=2))
+        print(f"[patch] quantization_config made loadable (quant_type={qc['quant_type']})")
+    else:
+        print("[patch] no embedded quantization_config; relying on modelopt_state.pth for restore")
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--format", choices=["fp8", "nvfp4"], default="nvfp4")
+    ap.add_argument("--serve-dir", required=True,
+                    help="existing assembled pipeline dir (source of VAE / tokenizers / model_index.json)")
+    ap.add_argument("--out-dir", required=True, help="new drop-in repo dir to create")
+    ap.add_argument("--cache", default=None, help="optional mto cache dir for a faster rebuild")
+    ap.add_argument("--gpu-mem-fraction", type=float, default=0.85)
+    args = ap.parse_args()
+
+    # 1. quantized transformer in memory (restore from cache if available, else rebuild)
+    model = try_restore_quantized(args.format, args.cache) if args.cache else None
+    if model is None:
+        model = build_quantized_transformer(args.format, args.gpu_mem_fraction)
+
+    # 2. enable round-trippable HF checkpointing, then save the transformer
+    enable_huggingface_checkpointing()
+    tdir = os.path.join(args.out_dir, "transformer")
+    os.makedirs(tdir, exist_ok=True)
+    print(f"[save] writing round-trippable transformer (+ modelopt_state.pth) -> {tdir}")
+    model.save_pretrained(tdir)
+    state_file = os.path.join(tdir, "modelopt_state.pth")
+    assert os.path.isfile(state_file), (
+        f"expected {state_file} to exist -- enable_huggingface_checkpointing() must run "
+        "before save_pretrained(); without modelopt_state.pth the repo won't load in diffusers"
+    )
+    ensure_loadable_config(tdir, args.format)
+
+    # 3. copy the rest of the pipeline (everything except transformer/) from the serve dir
+    print(f"[assemble] copying non-transformer components from {args.serve_dir}")
+    for name in os.listdir(args.serve_dir):
+        if name == "transformer":
+            continue
+        src = os.path.join(args.serve_dir, name)
+        dst = os.path.join(args.out_dir, name)
+        if os.path.isdir(src):
+            shutil.copytree(src, dst, dirs_exist_ok=True)
+        else:
+            shutil.copy2(src, dst)
+
+    print(f"[done] drop-in repo -> {args.out_dir}")
+    print(f"       verify:  python -i load_cosmos3_modelopt.py {args.out_dir}")
+
+
+if __name__ == "__main__":
+    main()

scheduler/scheduler_config.json

@@ -0,0 +1,33 @@
+{
+  "_class_name": "UniPCMultistepScheduler",
+  "_diffusers_version": "0.37.1",
+  "beta_end": 0.02,
+  "beta_schedule": "linear",
+  "beta_start": 0.0001,
+  "disable_corrector": [],
+  "dynamic_thresholding_ratio": 0.995,
+  "final_sigmas_type": "zero",
+  "flow_shift": 1.0,
+  "lower_order_final": true,
+  "num_train_timesteps": 1000,
+  "predict_x0": true,
+  "prediction_type": "flow_prediction",
+  "rescale_betas_zero_snr": false,
+  "sample_max_value": 1.0,
+  "shift_terminal": null,
+  "sigma_max": 200.0,
+  "sigma_min": 0.147,
+  "solver_order": 2,
+  "solver_p": null,
+  "solver_type": "bh2",
+  "steps_offset": 0,
+  "thresholding": false,
+  "time_shift_type": "exponential",
+  "timestep_spacing": "linspace",
+  "trained_betas": null,
+  "use_beta_sigmas": false,
+  "use_dynamic_shifting": false,
+  "use_exponential_sigmas": false,
+  "use_flow_sigmas": true,
+  "use_karras_sigmas": true
+}

serve_cosmos3_diffusers.py

@@ -0,0 +1,446 @@
+#!/usr/bin/env python
+"""
+Local generation server for your quantized Cosmos3-Super, built on the *validated*
+diffusers path (NOT vLLM-Omni).
+
+This reproduces, at startup, the exact in-memory model your streaming quantizer
+already rendered from successfully -- build empty on meta, insert weight-only
+quantizers, compress, stream the BF16 shards into compressed form -- then serves
+that model behind a tiny HTTP API. It does NOT reload the export_hf_checkpoint
+output (that unified format is for vLLM-Omni / TRT-LLM; diffusers round-trips a
+ModelOpt model via modelopt_state + state_dict, which is what --cache uses below).
+
+Nothing here is speculative: the model object served is the same one that produced
+cosmos3_super_<fmt>_validate.png.
+
+ENDPOINTS
+---------
+  GET  /health    -> readiness + which format is loaded
+  POST /generate  -> text -> still image (JSON body; returns PNG)
+  POST /animate   -> image -> video (multipart upload; returns MP4, or GIF if no
+                     mp4 encoder is installed)
+
+ENV / DEPS
+----------
+Run in the venv that has diffusers-from-git-main + modelopt + accelerate (your
+quantization venv, e.g. /home/prometheus/ModelOpt/.venv). Extra installs:
+
+    pip install fastapi uvicorn python-multipart      # python-multipart is REQUIRED for /animate
+    pip install imageio imageio-ffmpeg                # optional: mp4 output (else /animate returns GIF)
+
+USAGE
+-----
+    CUDA_VISIBLE_DEVICES=0 python serve_cosmos3_diffusers.py --format nvfp4
+    # faster restarts after the first boot (writes/reads a ~36 GB cache):
+    CUDA_VISIBLE_DEVICES=0 python serve_cosmos3_diffusers.py --format nvfp4 --cache ./cosmos3-cache
+
+Text -> still image:
+    curl -s -X POST http://localhost:8000/generate \
+        -H 'Content-Type: application/json' \
+        -d '{"prompt":"a robot arm on a workbench in a bright lab","num_inference_steps":50}' \
+        --output out.png
+
+Image -> video (upload the conditioning frame, so server-side paths never matter;
+`@` makes curl attach the file from YOUR current directory, and a shell ~ is expanded
+by the shell before curl runs):
+    curl -s -X POST http://localhost:8000/animate \
+        -F image=@out.png \
+        -F 'prompt=The robotic arm slowly lowers its gripper toward the objects and holds. Static camera.' \
+        -F num_frames=49 -F fps=24 \
+        --output clip.mp4
+
+Health:
+    curl -s http://localhost:8000/health
+"""
+
+import argparse
+import contextlib
+import gc
+import io
+import os
+import tempfile
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
+
+import torch
+from accelerate import init_empty_weights, load_checkpoint_in_model
+from accelerate.utils import get_max_memory, infer_auto_device_map
+from accelerate.utils.dataclasses import CustomDtype
+from huggingface_hub import snapshot_download
+from PIL import Image
+
+import modelopt.torch.quantization as mtq
+from diffusers import Cosmos3OmniTransformer
+from diffusers.utils import export_to_gif, export_to_video
+
+SRC_REPO = "nvidia/Cosmos3-Super"
+
+# --- hard-won config from the validated quantizer (inlined so this file stands alone) ---
+SPARE_SUBSTRINGS = [
+    "time_embedder", "proj_in", "proj_out", "lm_head", "embed", "norm", "audio_proj",
+]
+
+
+def _is_spare(name: str) -> bool:
+    return any(s in name for s in SPARE_SUBSTRINGS)
+
+
+def build_quant_cfg(fmt: str) -> dict:
+    if fmt == "fp8":
+        return {
+            "quant_cfg": {
+                "*weight_quantizer": {"num_bits": (4, 3), "axis": None, "enable": True},
+                "*input_quantizer": {"enable": False},
+                "*output_quantizer": {"enable": False},
+                "*softmax_quantizer": {"enable": False},
+            },
+            "algorithm": "max",
+        }
+    if fmt == "nvfp4":
+        import copy
+        base = getattr(mtq, "W4A16_NVFP4_CFG", None) or mtq.NVFP4_DEFAULT_CFG
+        cfg = copy.deepcopy(base)
+        # Bake weight-only INTO THE CONFIG: modelopt_state replays the config, not
+        # imperative .disable() calls made after quantize. NVFP4_DEFAULT_CFG ships with
+        # activation quantization enabled, so without this, a restored checkpoint comes
+        # back with ~1806 dynamic activation quantizers active (~10x slower per step).
+        # The drop-in loader re-disables as belt-and-braces, but the saved state should
+        # be correct on its own. (The FP8 dict below already does this.)
+        cfg.setdefault("quant_cfg", {})
+        cfg["quant_cfg"]["*input_quantizer"] = {"enable": False}
+        cfg["quant_cfg"]["*output_quantizer"] = {"enable": False}
+        cfg["quant_cfg"]["*softmax_quantizer"] = {"enable": False}
+        for s in SPARE_SUBSTRINGS:
+            cfg["quant_cfg"][f"*{s}*weight_quantizer"] = {"enable": False}
+        return cfg
+    raise ValueError(f"Unknown format: {fmt!r}")
+
+
+def enforce_weight_only_and_spare(model) -> tuple[int, int]:
+    n_spare = n_act = 0
+    for name, module in model.named_modules():
+        if not (name.endswith("_quantizer") and hasattr(module, "disable")):
+            continue
+        if name.endswith("weight_quantizer"):
+            if _is_spare(name.rsplit(".", 1)[0]):
+                module.disable()
+                n_spare += 1
+        else:
+            module.disable()
+            n_act += 1
+    return n_spare, n_act
+
+
+def compressed_device_map(model, gpu_mem_fraction: float = 0.85) -> dict:
+    max_memory = {k: v * gpu_mem_fraction for k, v in get_max_memory().items()}
+    no_split = set()
+    for name, module in model.named_modules():
+        if name.endswith((".layers.0", ".blocks.0", ".transformer_blocks.0")):
+            no_split.add(module.__class__.__name__)
+    special_dtypes = {}
+    for name, module in model.named_modules():
+        if (
+            hasattr(module, "weight")
+            and hasattr(module, "weight_quantizer")
+            and getattr(module.weight_quantizer, "is_enabled", True)
+            and not getattr(module.weight_quantizer, "fake_quant", True)
+        ):
+            nb = module.weight_quantizer.num_bits
+            if isinstance(nb, tuple):
+                nb = nb[0] + nb[1] + 1
+            special_dtypes[name + ".weight"] = CustomDtype.FP8 if nb == 8 else CustomDtype.INT4
+    return infer_auto_device_map(
+        model, max_memory=max_memory,
+        no_split_module_classes=list(no_split), special_dtypes=special_dtypes,
+    )
+
+
+def _materialize_residual_meta(model) -> int:
+    """Fill any leftover meta tensors (disabled-quantizer scratch) with zeros on GPU."""
+    n = 0
+    for _, module in model.named_modules():
+        for bn, buf in list(module._buffers.items()):
+            if buf is not None and getattr(buf, "is_meta", False):
+                module._buffers[bn] = torch.zeros(buf.shape, dtype=buf.dtype, device="cuda")
+                n += 1
+        for pn, par in list(module._parameters.items()):
+            if par is not None and getattr(par, "is_meta", False):
+                module._parameters[pn] = torch.nn.Parameter(
+                    torch.zeros(par.shape, dtype=par.dtype, device="cuda"), requires_grad=False
+                )
+                n += 1
+    return n
+
+
+def _transformer_dir() -> str:
+    local_root = snapshot_download(SRC_REPO, allow_patterns=["transformer/*"])
+    return os.path.join(local_root, "transformer")
+
+
+def build_quantized_transformer(fmt: str, gpu_mem_fraction: float = 0.85):
+    """The proven path: empty-on-meta -> quantize -> compress -> stream shards in."""
+    transformer_dir = _transformer_dir()
+    print(f"[build] empty transformer on meta from {transformer_dir}")
+    config = Cosmos3OmniTransformer.load_config(transformer_dir)
+    with init_empty_weights(include_buffers=False):
+        model = Cosmos3OmniTransformer.from_config(config)
+
+    print(f"[build] inserting weight-only {fmt} quantizers")
+    mtq.quantize(model, build_quant_cfg(fmt))
+    n_spare, n_act = enforce_weight_only_and_spare(model)
+    print(f"[build] weight-only: disabled {n_act} activation quantizers; {n_spare} spare weight layers")
+
+    print("[build] setting up compressed parameter shapes")
+    try:
+        mtq.compress(model, config=mtq.CompressConfig(quant_gemm=False))
+    except (AttributeError, TypeError):
+        mtq.compress(model)
+
+    print("[build] streaming BF16 shards into compressed form (slow step)")
+    load_checkpoint_in_model(
+        model, checkpoint=transformer_dir,
+        device_map=compressed_device_map(model, gpu_mem_fraction), dtype=torch.bfloat16,
+    )
+    fixed = _materialize_residual_meta(model)
+    if fixed:
+        print(f"[build] materialized {fixed} residual meta tensors")
+    return model
+
+
+# --- optional fast-restart cache (modelopt_state + weights, per ModelOpt docs) ----------
+def _cache_paths(cache_dir: str, fmt: str):
+    return (os.path.join(cache_dir, f"modelopt_state_{fmt}.pt"),
+            os.path.join(cache_dir, f"weights_{fmt}.pt"))
+
+
+def save_quantized(model, fmt: str, cache_dir: str) -> None:
+    try:
+        from modelopt.torch.opt import modelopt_state
+    except ImportError:
+        from modelopt.torch.opt.conversion import modelopt_state
+    os.makedirs(cache_dir, exist_ok=True)
+    state_path, weights_path = _cache_paths(cache_dir, fmt)
+    print(f"[cache] writing {state_path} + {weights_path} (large; one time)")
+    torch.save(modelopt_state(model), state_path)
+    torch.save(model.state_dict(), weights_path)
+
+
+def try_restore_quantized(fmt: str, cache_dir: str):
+    """Restore the compressed model from cache. Returns model or None (caller rebuilds)."""
+    state_path, weights_path = _cache_paths(cache_dir, fmt)
+    if not (os.path.isfile(state_path) and os.path.isfile(weights_path)):
+        return None
+    try:
+        try:
+            from modelopt.torch.opt import restore_from_modelopt_state
+        except ImportError:
+            from modelopt.torch.opt.conversion import restore_from_modelopt_state
+        print(f"[cache] restoring from {state_path}")
+        config = Cosmos3OmniTransformer.load_config(_transformer_dir())
+        with init_empty_weights(include_buffers=False):
+            model = Cosmos3OmniTransformer.from_config(config)
+        state = torch.load(state_path, map_location="cpu", weights_only=False)
+        restore_from_modelopt_state(model, state)  # replays quantize + compress structure
+        weights = torch.load(weights_path, map_location="cpu", weights_only=False)
+        model.load_state_dict(weights, strict=False, assign=True)
+        _materialize_residual_meta(model)
+        print("[cache] restore OK")
+        return model
+    except Exception as e:
+        import traceback
+        print(f"[cache] restore failed ({type(e).__name__}: {e}); falling back to full rebuild")
+        traceback.print_exc()
+        return None
+
+
+# --- pipeline assembly (mirrors the validated render_from_memory) -----------------------
+def make_pipeline(model, flow_shift: float = 3.0):
+    from diffusers import Cosmos3OmniPipeline
+    from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
+
+    model.to("cuda")
+    # dtype consistency: keep FP8/NVFP4 weights, but bring stray fp32 buffers to bf16 and
+    # cast time-embedder inputs at the fp32->bf16 boundary (the validated nudges).
+    for m in model.modules():
+        for bn, buf in list(m._buffers.items()):
+            if buf is not None and buf.dtype == torch.float32:
+                m._buffers[bn] = buf.to(torch.bfloat16)
+
+    def _cast_bf16(_m, args):
+        return tuple(
+            a.to(torch.bfloat16)
+            if torch.is_tensor(a) and a.is_floating_point() and a.dtype != torch.bfloat16 else a
+            for a in args
+        )
+
+    for name, m in model.named_modules():
+        if "time_embedder" in name and hasattr(m, "linear_1"):
+            m.register_forward_pre_hook(_cast_bf16)
+
+    pipe = Cosmos3OmniPipeline.from_pretrained(
+        SRC_REPO, transformer=model, torch_dtype=torch.bfloat16,
+        enable_safety_checker=False,  # local single-user server; revisit if exposing it
+    )
+    pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=flow_shift)
+    for name, comp in pipe.components.items():
+        if name != "transformer" and isinstance(comp, torch.nn.Module):
+            comp.to("cuda")
+    return pipe
+
+
+# --- HTTP server ------------------------------------------------------------------------
+import asyncio
+from fastapi import FastAPI, File, Form, UploadFile
+from fastapi.responses import Response
+from pydantic import BaseModel
+
+STATE: dict = {}
+_gen_lock = asyncio.Lock()  # one generation at a time on a single GPU
+
+
+# ---- text -> still image -------------------------------------------------------
+class GenRequest(BaseModel):
+    prompt: str
+    negative_prompt: str = ""
+    num_inference_steps: int = 50
+    guidance_scale: float = 4.0
+    height: int = 1024
+    width: int = 1024
+    num_frames: int = 1            # 1 = still image; >1 = video frames (heavier)
+    seed: int | None = 1234        # null -> random each call
+
+
+def _run_generation(req: GenRequest) -> bytes:
+    pipe = STATE["pipe"]
+    gen = torch.Generator(device="cuda").manual_seed(int(req.seed)) if req.seed is not None else None
+    with torch.inference_mode():
+        result = pipe(
+            prompt=req.prompt,
+            negative_prompt=req.negative_prompt,
+            num_frames=req.num_frames,
+            height=req.height,
+            width=req.width,
+            num_inference_steps=req.num_inference_steps,
+            guidance_scale=req.guidance_scale,
+            generator=gen,
+        )
+    img = result.video[0]  # PIL image for the first (or only) frame
+    buf = io.BytesIO()
+    img.save(buf, format="PNG")
+    del result
+    gc.collect()
+    torch.cuda.empty_cache()
+    return buf.getvalue()
+
+
+# ---- image -> video (i2v) ------------------------------------------------------
+def _run_i2v(pil_image, prompt, negative_prompt, num_frames, fps,
+             height, width, steps, guidance, seed) -> tuple[bytes, str]:
+    pipe = STATE["pipe"]
+    image = pil_image.convert("RGB")  # the pipeline resizes this to (height, width)
+    gen = torch.Generator(device="cuda").manual_seed(int(seed)) if seed >= 0 else None
+    with torch.inference_mode():
+        result = pipe(
+            prompt=prompt, negative_prompt=negative_prompt,
+            image=image, num_frames=num_frames, fps=fps,
+            height=height, width=width,
+            num_inference_steps=steps, guidance_scale=guidance,
+            enable_safety_check=False, generator=gen, output_type="pil",
+        )
+    frames = result.video  # list of PIL frames
+    try:
+        with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tf:
+            path = tf.name
+        export_to_video(frames, path, fps=int(round(fps)))
+        media = "video/mp4"
+    except Exception:  # no mp4 backend installed -> GIF (PIL-only, always works)
+        with tempfile.NamedTemporaryFile(suffix=".gif", delete=False) as tf:
+            path = tf.name
+        export_to_gif(frames, path)
+        media = "image/gif"
+    data = open(path, "rb").read()
+    os.remove(path)
+    del result
+    gc.collect()
+    torch.cuda.empty_cache()
+    return data, media
+
+
+@contextlib.asynccontextmanager
+async def lifespan(app: FastAPI):
+    fmt = STATE["fmt"]
+    cache_dir = STATE.get("cache_dir")
+    model = None
+    if cache_dir:
+        model = try_restore_quantized(fmt, cache_dir)
+    if model is None:
+        model = build_quantized_transformer(fmt, STATE["gpu_mem_fraction"])
+        if cache_dir:
+            try:
+                save_quantized(model, fmt, cache_dir)
+            except Exception as e:
+                print(f"[cache] save failed ({type(e).__name__}: {e}); continuing without cache")
+    STATE["pipe"] = make_pipeline(model, STATE["flow_shift"])
+    print(f"[ready] serving {fmt.upper()} Cosmos3-Super on diffusers")
+    yield
+    STATE.clear()
+
+
+app = FastAPI(lifespan=lifespan)
+
+
+@app.get("/health")
+async def health():
+    return {"status": "ok" if "pipe" in STATE else "loading", "format": STATE.get("fmt")}
+
+
+@app.post("/generate")
+async def generate(req: GenRequest):
+    async with _gen_lock:
+        loop = asyncio.get_running_loop()
+        png = await loop.run_in_executor(None, _run_generation, req)
+    return Response(content=png, media_type="image/png")
+
+
+@app.post("/animate")
+async def animate(
+    image: UploadFile = File(...),
+    prompt: str = Form(...),
+    negative_prompt: str = Form(""),
+    num_frames: int = Form(49),         # ~2.04s @ 24fps; 4n+1 maps cleanly to the VAE's 4x temporal compression
+    fps: float = Form(24.0),            # native framerate; it conditions duration + audio length, so keep 24
+    height: int = Form(1024),
+    width: int = Form(1024),
+    num_inference_steps: int = Form(35),  # video default (the still path uses 50)
+    guidance_scale: float = Form(6.0),    # video default (the still path uses 4.0)
+    seed: int = Form(1234),               # pass -1 for a random clip each call
+):
+    pil = Image.open(io.BytesIO(await image.read()))
+    async with _gen_lock:
+        loop = asyncio.get_running_loop()
+        data, media = await loop.run_in_executor(
+            None, _run_i2v, pil, prompt, negative_prompt, num_frames, fps,
+            height, width, num_inference_steps, guidance_scale, seed,
+        )
+    return Response(content=data, media_type=media)
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--format", choices=["fp8", "nvfp4"], default="nvfp4")
+    ap.add_argument("--host", default="0.0.0.0")
+    ap.add_argument("--port", type=int, default=8000)
+    ap.add_argument("--flow-shift", type=float, default=3.0)
+    ap.add_argument("--gpu-mem-fraction", type=float, default=0.85)
+    ap.add_argument("--cache", default=None,
+                    help="Dir for a fast-restart cache. First boot rebuilds + writes it; "
+                         "later boots restore from it. Any restore error -> full rebuild.")
+    args = ap.parse_args()
+
+    STATE.update(
+        fmt=args.format, flow_shift=args.flow_shift,
+        gpu_mem_fraction=args.gpu_mem_fraction, cache_dir=args.cache,
+    )
+
+    import uvicorn
+    uvicorn.run(app, host=args.host, port=args.port)

sound_tokenizer/config.json

@@ -0,0 +1,64 @@
+{
+    "model_type": "autoencoder_v2",
+    "sampling_rate": 48000,
+    "stereo": true,
+    "use_wav_as_input": true,
+    "normalize_volume": true,
+    "hop_size": 1920,
+    "input_channels": 1,
+    "enc_type": "spec_convnext",
+    "enc_dim": 192,
+    "enc_intermediate_dim": 768,
+    "enc_num_layers": 12,
+    "enc_num_blocks": 2,
+    "enc_n_fft": 64,
+    "enc_hop_length": 16,
+    "enc_latent_dim": 128,
+    "enc_c_mults": [
+        1,
+        2,
+        4
+    ],
+    "enc_strides": [
+        4,
+        5,
+        6
+    ],
+    "enc_identity_init": false,
+    "enc_use_snake": true,
+    "dec_type": "oobleck",
+    "dec_dim": 320,
+    "dec_c_mults": [
+        1,
+        2,
+        4,
+        8,
+        16
+    ],
+    "dec_strides": [
+        2,
+        4,
+        5,
+        6,
+        8
+    ],
+    "dec_use_snake": true,
+    "dec_final_tanh": false,
+    "dec_out_channels": 2,
+    "dec_anti_aliasing": false,
+    "dec_use_nearest_upsample": false,
+    "dec_use_tanh_at_final": false,
+    "bottleneck_type": "vae",
+    "bottleneck": {
+        "type": "vae"
+    },
+    "activation": "snakebeta",
+    "snake_logscale": true,
+    "anti_aliasing": false,
+    "use_cuda_kernel": false,
+    "causal": false,
+    "padding_mode": "zeros",
+    "vocoder_input_dim": 64,
+    "latent_mean": null,
+    "latent_std": null
+}

sound_tokenizer/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d4c61cde38acfb0cad9048a140c3533750277a8462b19dc08450d9fe1ad9879
+size 1892409600

text_tokenizer/added_tokens.json

@@ -0,0 +1,28 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

text_tokenizer/chat_template.jinja

@@ -0,0 +1,120 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' }}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- for message in messages %}
+    {%- if message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content_item in message.content %}
+                {%- if 'text' in content_item %}
+                    {{- content_item.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and message.content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|vision_start|><|image_pad|><|vision_end|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|vision_start|><|video_pad|><|vision_end|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}