Document Cosmos3-Nano S1/S3 checkpoints and evaluation

Adds Cosmos3-Nano S1/S3 model-card coverage, inference instructions, shared training/evaluation descriptions, licensing, and completed DL3DV evaluation results while preserving the Wan entries. The verified S1/S3 checkpoint bundle, metadata, checksum manifest and licenses will be added to this PR before release. Source PR: https://github.com/nv-tlabs/ArtiFixer/pull/22.

README.md

@@ -7,14 +7,15 @@ library_name: artifixer
 ## Description:
  
 ArtiFixer is a few-step causal auto-regressive model that enhances and extends 3D reconstruction. The related source code provides implementations for training, evaluation, and inference, supporting various stages including bidirectional training, diffusion forcing, and Self-Forcing-style DMD distillation.  
-ArtiFixer was developed by NVIDIA (Spatial Intelligence Lab) and is released in two variants: a 14B variant based on Wan2.1's 14B model, and a lighter-weight 1.3B variant based on Wan2.1's 1.3B model that trades some reconstruction quality for significantly higher throughput and lower training and inference compute.  
-In addition to the distilled few-step causal auto-regressive models, this repository also provides the bidirectional diffusion **teacher** checkpoints from which each released model was distilled (`artifixer-s1-14b.pt` and `artifixer-s1-1.3b.pt`, the stage-1 bidirectional training outputs). The teachers use full bidirectional attention with standard multi-step diffusion sampling instead of few-step causal auto-regressive rollout. They are provided for research on distillation, sampling, and long-rollout behavior, and as the starting point for reproducing the diffusion-forcing and DMD distillation stages.  
+ArtiFixer was developed by NVIDIA (Spatial Intelligence Lab) and provides distilled causal models and their bidirectional S1 teachers in three variants: Wan2.1 14B, Wan2.1 1.3B, and Cosmos3-Nano. The Wan2.1 1.3B variant trades some reconstruction quality for higher throughput and lower compute.  
+The distilled stage-3 (S3) checkpoints are [`artifixer-14b.pt`](./artifixer-14b.pt), [`artifixer-1.3b.pt`](./artifixer-1.3b.pt), and [`artifixer-cosmos3-nano.pt`](./artifixer-cosmos3-nano.pt).
+In addition to the distilled few-step causal auto-regressive models, this repository also provides the bidirectional diffusion **teacher** checkpoints from which each released model was distilled ([`artifixer-s1-14b.pt`](./artifixer-s1-14b.pt), [`artifixer-s1-1.3b.pt`](./artifixer-s1-1.3b.pt), and [`artifixer-s1-cosmos3-nano.pt`](./artifixer-s1-cosmos3-nano.pt), the stage-1 bidirectional training outputs). The teachers use full bidirectional attention with standard multi-step diffusion sampling instead of few-step causal auto-regressive rollout. They are provided for research on distillation, sampling, and long-rollout behavior, and as the starting point for reproducing the diffusion-forcing and DMD distillation stages.  
 _This model is for research and development only._
  
 ### License/Terms of Use:
  
 GOVERNING DOWNLOAD TERMS: Use of the model is governed by the [NVIDIA License](https://developer.download.nvidia.com/licenses/NVIDIA-OneWay-Noncommercial-License-22Mar2022.pdf?t=eyJscyI6ImdzZW8iLCJsc2QiOiJodHRwczovL3d3dy5nb29nbGUuY29tLyIsIm5jaWQiOiJzby15b3V0LTg3MTcwMS12dDQ4In0=).  
-ADDITIONAL INFORMATION: The Wan 2.1 14B and 1.3B base models are governed by the [Apache License, Version 2.0.](https://www.apache.org/licenses/LICENSE-2.0)
+ADDITIONAL INFORMATION: The Wan 2.1 14B and 1.3B base models are governed by the [Apache License, Version 2.0.](https://www.apache.org/licenses/LICENSE-2.0) The [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano) base model is governed by the [Open MDW 1.1 License](https://openmdw.ai/license/1-1/).
  
 ### Deployment Geography:
  
@@ -26,7 +27,7 @@ Developers and researchers working on 3D reconstruction, diffusion models, and a
  
 ### Release Date:
  
-**Other:** Hugging Face: 06/04/2026 (14B variant), 07/21/2026 (1.3B variant), and 09/2026 (bidirectional teacher checkpoints) via https://research.nvidia.com/labs/sil/projects/artifixer/
+**Other:** Hugging Face: 06/04/2026 (14B variant), 07/21/2026 (1.3B variant), and 09/2026 (Wan bidirectional teacher checkpoints) via https://research.nvidia.com/labs/sil/projects/artifixer/
  
 ## Reference(s):
  
@@ -35,10 +36,18 @@ Developers and researchers working on 3D reconstruction, diffusion models, and a
 ## Model Architecture:
  
 **Architecture Type:** Transformer  
-**Network Architecture:** ArtifixerTransformer (built on Wan2.1's WanTransformer3DModel)  
-**This model was developed based on Wan-AI/Wan2.1-T2V-14B-Diffusers (14B variant) and Wan-AI/Wan2.1-T2V-1.3B-Diffusers (1.3B variant).**  
-**Number of model parameters:** 14B variant: ~16.9B trainable (16,910,955,584); 1.3B variant: ~1.68B trainable (1,678,749,760). The teacher checkpoints share the architecture and parameter counts of their respective variants.
- 
+**Network Architecture:** ArtifixerTransformer (built on Wan2.1's WanTransformer3DModel) and ArtifixerCosmos3Transformer (built on Cosmos3OmniTransformer)  
+**The variants are based on Wan-AI/Wan2.1-T2V-14B-Diffusers, Wan-AI/Wan2.1-T2V-1.3B-Diffusers, and nvidia/Cosmos3-Nano, respectively.**  
+**Number of model parameters:** 14B variant: ~16.9B trainable (16,910,955,584); 1.3B variant: ~1.68B trainable (1,678,749,760); Cosmos3-Nano: ~15.8B (15,802,891,556). The teacher checkpoints share the architecture and parameter counts of their respective variants.
+ 
+**Cosmos3-Nano architecture:** ArtiFixer's Cosmos3 transformer is based on
+`nvidia/Cosmos3-Nano` at revision `23314034b82f46b45339035dba67c3ee9bbcb8ba`.
+It uses token-ID prompts and
+SCoPE camera conditioning, with reference and target images in shared
+self-attention. The exported checkpoints contain the transformer weights for
+image-sequence reconstruction. The VAE, tokenizer, and configuration are loaded
+separately from the pinned base-model revision.
+
 ## Input:
  
 **Input Type(s):** Image, text  
@@ -51,7 +60,7 @@ Developers and researchers working on 3D reconstruction, diffusion models, and a
 **Output Type(s):** Image  
 **Output Format:** RGB (Red, Green, Blue)  
 **Output Parameters:** Two-Dimensional (2D) image frames  
-**Other Properties Related to Output:** Generates enhanced images via a few-step causal auto-regressive diffusion model. At 4 denoising steps on a single GPU, the 14B variant reaches 8.36 FPS and the 1.3B variant 34.38 FPS; with 1 denoising step and 4-GPU context parallelism, the 1.3B variant reaches up to 101.77 FPS. The bidirectional teacher checkpoints instead run standard multi-step diffusion sampling with full bidirectional attention; the FPS figures above apply to the distilled causal auto-regressive models only.
+**Other Properties Related to Output:** Generates enhanced images via a few-step causal auto-regressive diffusion model. At 4 denoising steps on a single GPU, the 14B variant reaches 8.36 FPS and the 1.3B variant 34.38 FPS; with 1 denoising step and 4-GPU context parallelism, the 1.3B variant reaches up to 101.77 FPS. These speed figures describe the Wan variants. The bidirectional teacher checkpoints instead run standard multi-step diffusion sampling with full bidirectional attention; the FPS figures above apply to the distilled causal auto-regressive models only.
  
 Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
  
@@ -65,7 +74,7 @@ The integration of foundation and fine-tuned models into AI systems requires add
  
 ## Model Version(s):
  
-ArtiFixer 14B v1.0, ArtiFixer 1.3B v1.0, ArtiFixer 14B Teacher v1.0, ArtiFixer 1.3B Teacher v1.0  
+ArtiFixer 14B v1.0, ArtiFixer 1.3B v1.0, ArtiFixer 14B Teacher v1.0, ArtiFixer 1.3B Teacher v1.0, ArtiFixer Cosmos3-Nano v1.0, ArtiFixer Cosmos3-Nano Teacher v1.0  
 The teacher checkpoints are the stage-1 bidirectional models from which the corresponding released causal auto-regressive models were distilled.  
 ArtiFixer integrates with PyTorch and requires CUDA environments. It uses Dockerfiles for CUDA 12 and 13, supporting both x86_64 and aarch64 architectures. The model can be run using torchrun (or accelerate) with multi-GPU setups and requires dependencies such as flash-attn, accelerate, diffusers, and transformers.
  
@@ -78,7 +87,7 @@ ArtiFixer integrates with PyTorch and requires CUDA environments. It uses Docker
 **Text Training Data Size:** Less than a Billion Tokens  
 **Data Collection Method by dataset:** Hybrid: Automated/Synthetic  
 **Labeling Method by dataset:** Automated  
-**Properties (Quantity, Dataset Descriptions, Sensor(s)):** Multimodal dataset combining 3D reconstruction data from DL3DV-10K (the DL3DV-ALL-960P release) with text captions. Includes sparse 3D point clouds, camera parameters, and RGB images. Camera poses are estimated with COLMAP, reconstructions are produced with 3DGUT (MCMC densification, via the 3DGRUT library), text captions are generated with a vision-language model (Qwen3-VL-30B-A3B-Instruct), and metric scale is estimated with MoGe. The dataset supports training for 3D reconstruction tasks with both real and generated data. Both variants are trained on the same dataset.
+**Properties (Quantity, Dataset Descriptions, Sensor(s)):** Multimodal dataset combining 3D reconstruction data from DL3DV-10K (the DL3DV-ALL-960P release) with text captions. Includes sparse 3D point clouds, camera parameters, and RGB images. Camera poses are estimated with COLMAP, reconstructions are produced with 3DGUT (MCMC densification, via the 3DGRUT library), text captions are generated with a vision-language model (Qwen3-VL-30B-A3B-Instruct), and metric scale is estimated with MoGe. The dataset supports training for 3D reconstruction tasks with both real and generated data. The Wan2.1 14B, Wan2.1 1.3B, and Cosmos3-Nano variants are trained on this dataset.
  
 ### Testing Dataset:
  
@@ -88,7 +97,7 @@ ArtiFixer integrates with PyTorch and requires CUDA environments. It uses Docker
  
 ### Evaluation Dataset:
  
-All numbers below are measured with the released causal auto-regressive checkpoints (`artifixer-14b.pt`, `artifixer-1.3b.pt`). Artifact removal on the Nerfbusters and DL3DV benchmarks (Difix3D+ protocol; NB = Nerfbusters):
+The tables identify the evaluated model variant and inference method. Artifact removal on the Nerfbusters and DL3DV benchmarks uses the released Wan causal auto-regressive checkpoints (`artifixer-14b.pt`, `artifixer-1.3b.pt`) with the Difix3D+ protocol (NB = Nerfbusters):
  
 | Method | NB PSNR↑ | NB SSIM↑ | NB LPIPS↓ | NB FID↓ | DL3DV PSNR↑ | DL3DV SSIM↑ | DL3DV LPIPS↓ | DL3DV FID↓ |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
@@ -112,6 +121,23 @@ Novel-content generation on DL3DV (our evaluation split), direct ArtiFixer:
 | 1.3B | 19.12 | 0.629 | 0.332 | 16.24 |
 | 14B | 20.27 | 0.657 | 0.301 | 12.33 |
  
+Cosmos3-Nano checkpoint evaluation on DL3DV (our split), direct ArtiFixer:
+
+| Checkpoint | Sampling | PSNR↑ | SSIM↑ | LPIPS↓ | FID↓ |
+| --- | --- | --- | --- | --- | --- |
+| Cosmos3-Nano S1 (step 25,000) | Bidirectional, 50 steps, CFG 5 | 19.17 | 0.629 | 0.359 | 13.47 |
+| Cosmos3-Nano S3 (EMA700) | KV cache, 4 steps | 19.58 | 0.629 | 0.348 | 11.75 |
+
+These results use six reference views, 28 scenes, and 9,619 held-out frames.
+PSNR, SSIM, and LPIPS are averaged over frames; FID is computed over the pooled
+frames. LPIPS uses standard VGG normalization with inputs in `[-1, 1]`. The historical Wan
+measurements above retain their original evaluation protocol.
+
+Both exported checkpoints were evaluated in BF16 on H100 80 GB GPUs, with four
+GPUs processing independent samples. S1 uses chunks of up to 81 source frames;
+S3 rolls out the full trajectory with block/window/sink sizes of 7/21/7 latent
+frames. Both use seed 42 and the pinned Cosmos3-Nano base-model revision.
+
 Additional results — Tanks & Temples and multi-view consistency (supplement) — are reported in the [paper](https://research.nvidia.com/labs/sil/projects/artifixer/assets/paper.pdf).
  
 **Data Collection Method by dataset:** Hybrid: Automated/Human  
@@ -123,6 +149,86 @@ Additional results — Tanks & Temples and multi-view consistency (supplement)
 **Acceleration Engine:** FlashAttention (FA3/FA4); PyTorch SDPA (cuDNN) fallback on Ampere  
 **Test Hardware:** NVIDIA A100 80GB, NVIDIA H100, NVIDIA GB200 (Blackwell)
  
+## Cosmos3-Nano checkpoints and usage
+
+| Checkpoint | Training stage | Sampling |
+| --- | --- | --- |
+| [`artifixer-s1-cosmos3-nano.pt`](./artifixer-s1-cosmos3-nano.pt) | S1 bidirectional teacher, step 25,000 | Bidirectional, 50 steps, text guidance 5 |
+| [`artifixer-cosmos3-nano.pt`](./artifixer-cosmos3-nano.pt) | S3 DMD, EMA step 700 | Causal KV cache, 4 steps; block/window/sink = 7/21/7 latent frames |
+
+S1 is the teacher used to initialize the diffusion-forcing stage preceding S3
+DMD distillation. Both variants are trained on DL3DV-10K (DL3DV-ALL-960P),
+using sparse-view reconstructions, calibrated cameras, and text captions.
+The [training guide](https://github.com/nv-tlabs/ArtiFixer/blob/main/docs/training.md)
+describes the three-stage recipe and the checkpoint handoffs.
+
+Each FP32 transformer checkpoint is approximately 63.2 GB (58.9 GiB). BF16
+runtime transformer weights alone occupy 31.6 GB (29.4 GiB), before the VAE,
+activations, and KV cache. Use the matching ArtiFixer source and Docker
+environment from the [repository](https://github.com/nv-tlabs/ArtiFixer), with
+host FFmpeg supplied as described in the
+[setup guide](https://github.com/nv-tlabs/ArtiFixer/blob/main/docs/setup.md).
+
+Download both checkpoints and their required metadata sidecars:
+
+```bash
+export CHECKPOINT_ROOT=/data/artifixer-checkpoints
+hf download nvidia/ArtiFixer \
+    artifixer-cosmos3-nano.pt artifixer-cosmos3-nano.metadata.json \
+    artifixer-s1-cosmos3-nano.pt artifixer-s1-cosmos3-nano.metadata.json \
+    licenses/NOTICE.txt licenses/OpenMDW-1.1.txt \
+    licenses/NVIDIA-OneWay-Noncommercial-License-22Mar2022.pdf \
+    SHA256SUMS --local-dir "$CHECKPOINT_ROOT"
+(cd "$CHECKPOINT_ROOT" && sha256sum --check SHA256SUMS)
+```
+
+Prepare a calibrated COLMAP scene using the repository's input-layout guide.
+Use the actual source FPS (60 in this DL3DV example), a fresh output root,
+and Cosmos token-ID prompts:
+
+```bash
+export COLMAP_SCENE=/data/colmap-scenes/my_scene
+export SCENE_ROOT=/data/artifixer-prep/my_scene-cosmos3
+python -m data_processing.prepare_colmap_artifixer_inputs \
+    --colmap_dir "$COLMAP_SCENE" --output_root "$SCENE_ROOT" \
+    --captioning_model_id Qwen/Qwen3-VL-4B-Instruct \
+    --prompt_format cosmos3_omni_token_ids --dataset_fps 60
+
+python -m model_eval.run_inference \
+    --evalset reconstructed_colmap \
+    --checkpoint "$CHECKPOINT_ROOT/artifixer-cosmos3-nano.pt" \
+    --model_id nvidia/Cosmos3-Nano \
+    --model_revision 23314034b82f46b45339035dba67c3ee9bbcb8ba \
+    --split_path "$SCENE_ROOT/split.json" --render_trajectory all_frames \
+    --save_dir /data/artifixer-corrected --render_name cosmos3-nano-s3 \
+    --inference_pipeline kv_cache --num_inference_steps 4 \
+    --frames_per_block 7 --local_attn_size 21 --sink_size 7 \
+    --camera_pose_alignment last --seed 42
+```
+
+For the S1 teacher, use the same prepared inputs with bidirectional sampling:
+
+```bash
+python -m model_eval.run_inference \
+    --evalset reconstructed_colmap \
+    --checkpoint "$CHECKPOINT_ROOT/artifixer-s1-cosmos3-nano.pt" \
+    --model_id nvidia/Cosmos3-Nano \
+    --model_revision 23314034b82f46b45339035dba67c3ee9bbcb8ba \
+    --split_path "$SCENE_ROOT/split.json" --render_trajectory all_frames \
+    --save_dir /data/artifixer-corrected --render_name cosmos3-nano-s1 \
+    --inference_pipeline bidirectional --num_inference_steps 50 \
+    --text_guidance_scale 5 --bidirectional_chunk_size 81 \
+    --camera_pose_alignment last --seed 42
+```
+
+Keep each metadata sidecar beside its weights. The camera-pose alignment is
+`last`; camera rays use the first target camera as their fixed origin. The model
+profile selects flow shift 5 and RoPE FPS 24; source FPS controls prompt timing.
+Wan T5 embedding files are incompatible with Cosmos token-ID prompts. Use a new
+render name for each setting to avoid reusing completed outputs. Text guidance
+is supported by the S1 bidirectional pipeline; S3 KV-cache sampling does not use
+text CFG.
+
 ## Ethical Considerations:
  
 NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.