Create README.md

README.md

@@ -0,0 +1,195 @@
+# **HMAR: Pretrained Multi-Scale Autoregressive Image Generation Models**
+
+[**Code**](https://github.com/NVlabs/HMAR) | [**Paper**](https://arxiv.org/abs/2506.04421) | [**Paper Website**](https://research.nvidia.com/labs/dir/hmar)
+
+# Model Overview
+
+## Description:
+
+Visual AutoRegressive modeling (VAR) has shown promise in bridging the speed and quality gap between autoregressive image models and diffusion models. VAR reformulates autoregressive modeling by decomposing an image into successive resolution scales. During inference, an image is generated by predicting all the tokens in the next (higher-resolution) scale, conditioned on all tokens in all previous (lower-resolution) scales. However, this formulation suffers from reduced image quality due to parallel generation of all tokens in a resolution scale; has sequence lengths scaling superlinearly in image resolution; and requires retraining to change the sampling schedule. 
+
+We introduce Hierarchical Masked AutoRegressive modeling (**HMAR**), a new image generation algorithm that alleviates these issues using next-scale prediction and masked prediction to generate high-quality images with fast sampling. HMAR reformulates next-scale prediction as a Markovian process, wherein prediction of each resolution scale is conditioned only on tokens in its immediate predecessor instead of the tokens in all predecessor resolutions. When predicting a resolution scale, HMAR uses a controllable multi-step masked generation procedure to generate a subset of the tokens in each step. On ImageNet 256x256 and 512x512 benchmarks, HMAR models match or outperform parameter-matched VAR, diffusion, and autoregressive baselines. We develop efficient IO-aware block-sparse attention kernels that allow HMAR to achieve faster training and inference times over VAR by over 2.5x and 1.75x respectively, as well as over 3x lower inference memory footprint. Finally HMAR yields additional flexibility over VAR; its sampling schedule can be changed without further training, and it can be applied to image editing tasks in a zero-shot manner.
+
+This model is for research and development/non-commercial use only. 
+
+**Model Developer**: NVIDIA
+
+## Model Versions
+
+We release 4 trained checkpoints for models of different sizes: `hmar-d16`, `hmar-d20`, `hmar-d24` and `hmar-d30` with 0.46B, 0.84B, 1.3B and 2.4B trainable parameters, respectively.
+
+- [hmar-d`N`] Given a ImageNet class label as input (from 0 to 999), the model produces an image that belongs to that class.
+
+### License:
+
+This model is released under the  [NVIDIA One-Way Noncommercial License (NSCLv1)](https://github.com/NVlabs/HMAR/blob/main/LICENSE). For a custom license, please contact [cosmos-license@nvidia.com](mailto:cosmos-license@nvidia.com).
+
+Under the NVIDIA One-Way Noncommercial License (NSCLv1), NVIDIA confirms:
+
+* Models are not for commercial use.
+* NVIDIA does not claim ownership to any outputs generated using the Models or Derivative Models.
+
+### Deployment Geography:
+
+Global
+
+### Use Case:
+
+Conditional Image Generation: Generation of images conditioned on a class label from the ImageNet dataset.
+
+### Release Date:
+
+* Github: [07/08/2025](https://github.com/NVlabs/HMAR)
+* Huggingface: [07/08/2025](https://huggingface.co/nvidia/HMAR/tree/main)
+
+## Model Architecture:
+
+*Architecture Type:* Transformer
+
+*Network Architecture:* Block-wise attention DiT
+
+This model was developed based on [VAR](https://github.com/FoundationVision/VAR).
+
+## Input
+
+  **Input Type(s)**: Class label (integer between 0 and 999)
+
+   **Input Format(s)**:
+   * Class label: Integer
+
+  **Input Parameters**:
+  * Class label: One-dimensional (1D)
+  
+  **Other Properties Related to Input**:
+  * The sampling configs can be modified at config/sampling/hmar-d30.yaml
+  * The number of masked sampling steps can be changed at utils/sampling_arg_util.py
+
+## Output
+
+ **Output Type**: Image 
+
+ **Output Format**: PNG
+
+ **Output Parameters**: Image: Two-dimensional (2D)
+
+ **Other Properties Related to Output:** The generated images are RGB images of size 256x256.
+
+ * Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br>
+
+## Software Integration
+
+**Runtime Engine(s):**
+
+Not Applicable (N/A)]
+
+**Supported Hardware Microarchitecture Compatibility:**
+
+* NVIDIA Blackwell
+* NVIDIA Hopper
+
+**Note**: We have only tested doing inference with BF16 precision.
+
+**Operating System(s):**
+
+* Linux (We have not tested on other operating systems.)
+
+
+# Usage
+
+See the [HMAR repository](https://github.com/nvlabs/hmar) for details.
+
+# Training, Testing, and Evaluation Datasets: <br>
+
+We use the [ImageNet dataset](https://www.image-net.org/) in our experiments –both for training, testing,  and evaluation.  ImageNet is a widely used dataset that spans 1000 object classes and contains 1,281,167 training images, 50,000 validation images and 100,000 test images. Each data sample is composed of an image-label pair.
+
+**The total size (in number of data points):** 1,431,167 <br>
+**Total number of datasets:** 1 <br>
+**Dataset partition:** Training [89.5 %], testing  [7 %], validation [3.5 %] <br>
+
+## Training Dataset: Official ImageNet training dataset
+
+*Link:*  https://www.image-net.org <br>
+
+## Test Dataset: Official ImageNet test dataset
+
+*Link:*  https://www.image-net.org <br>
+
+## Evaluation Dataset: Official ImageNet validation dataset
+
+*Link:*  https://www.image-net.org <br>
+
+# Evaluation
+
+Please see our [technical paper](https://arxiv.org/abs/2506.04421) for detailed evaluations.
+
+## Inference:
+**Acceleration Engine:** PyTorch, flash attention <br>
+**Test Hardware:** H100, A100, GB200 <br>
+* Minimum 1 GPU cards, multi nodes require Infiniband / ROCE connection <br>
+
+## Ethical Considerations
+
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+
+Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.
+
+For more detailed information on ethical considerations for this model, please see the Explainability, Bias, Safety & Security, and Privacy subcards below. 
+
+Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
+
+### Plus Plus (++) Promise
+
+We value you, the datasets, the diversity they represent, and what we have been entrusted with. This model and its associated data have been:
+
+* Verified to comply with current applicable disclosure laws, regulations, and industry standards.
+* Verified to comply with applicable privacy labeling requirements.
+* Annotated to describe the collector/source (NVIDIA or a third-party).
+* Characterized for technical limitations.
+* Reviewed to ensure proper disclosure is accessible to, maintained for, and in compliance with NVIDIA data subjects and their requests.
+* Reviewed before release.
+* Tagged for known restrictions and potential safety implications.
+
+### Bias
+
+| Field                                                                                                                                                            | Response |
+| :--------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------- |
+| Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing: | None     |
+| Measures taken to mitigate against unwanted bias:                                                                                                                | None     |
+
+### Explainability
+
+| Field                                                     | Response                                                                                                             |
+| :-------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------- |
+| Intended Application & Domain:                            | Image Generation                                                                                                |
+| Model Type:                                               | Transformer                                                                                                          |
+| Intended Users:                                           | Research                                                                                              |
+| Output:                                                   | Image                                                                                                                 |
+| Describe how the model works:                             | Generates images based on a class label from ImageNet                                                          |
+| Technical Limitations:                                    | Due to the stochastic nature of the model, it may not correctly follow the label on which the generation is conditioned at times.                                     |
+| Verified to have met prescribed NVIDIA quality standards: | Yes                                                                                                                  |
+| Performance Metrics:                                      | . We report multiple metrics for HMAR used for conditional image generation models. These are FID, IS, Precision and Recall metrics –see results table above. In addition, we perform human verification of the generated outputs to validate image quality and prompt following                                                                            |
+| Potential Known Risks:                                    | None Known |
+| Licensing:                                                | [NVIDIA One-Way Noncommercial License (NSCLv1)](https://github.com/NVlabs/HMAR/blob/main/LICENSE)   |
+
+### Privacy
+
+Field                                                                                                                              |  Response
+:----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
+Generatable or reverse engineerable personal data?                                                     |  No
+Personal data used to create this model?                                                                       | No
+How often is dataset reviewed?                                                                                                     |  Before Release 
+Is there provenance for all datasets used in training?                                                                                |  Not Applicable. Only externally-sourced data was used.
+Does data labeling (annotation, metadata) comply with privacy laws?                                                                |  Yes. 
+Is data compliant with data subject requests for data correction or removal, if such a request was made?                           |  No, not possible with externally-sourced data.
+Applicable Privacy Policy        | https://www.nvidia.com/en-us/about-nvidia/privacy-policy/ =
+
+
+
+### Safety
+
+| Field                                           | Response                                                                                                                                                                                                                                                                                                                             |
+| :---------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Model Application(s):                           | Conditional Image generation                                                                                                                                                                                                                                                                                                                     |
+| Describe the life critical impact (if present). | None Known                                                                                                                                                                                                                                                                                                                           |
+| Use Case Restrictions:                          | [NVIDIA One-Way Noncommercial License (NSCLv1)](https://github.com/NVlabs/HMAR/blob/main/LICENSE)                                                                                                                                                                                                                   |
+| Model and dataset restrictions:                 | The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development.  Restrictions enforce dataset access during training, and dataset license constraints adhered to. Model checkpoints are made available on Hugging Face, and may become available on cloud providers' model catalog. |