Update README.md
Browse files
README.md
CHANGED
|
@@ -16,6 +16,8 @@ By [Justin Deschenaux](https://jdeschena.github.io), [Lan Tran](https://github.c
|
|
| 16 |
|
| 17 |
[](https://arxiv.org/abs/2506.10892v1)
|
| 18 |
[](hhttps://huggingface.co/jdeschena/pgm)
|
|
|
|
|
|
|
| 19 |
|
| 20 |
**TL;DR: Partition Generative Models (PGMs) speed up parallel generation by partitioning tokens and using sparse attention instead of masking.**
|
| 21 |
|
|
@@ -23,22 +25,24 @@ By [Justin Deschenaux](https://jdeschena.github.io), [Lan Tran](https://github.c
|
|
| 23 |
<img src="https://jdeschena.github.io/pgm/static/images/pgm_vs_mgm.jpg" width="60%">
|
| 24 |
</div>
|
| 25 |
|
| 26 |
-
##
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
We release the raw checkpoints (distilled/undistilled) trained on OpenWebText (1M steps) and ImageNet (500k steps) on [Huggingface 🤗](https://huggingface.co/jdeschena/pgm). You need to download them from there if you want to run the evaluations.
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|
| 34 |
|
| 35 |
-
|
| 36 |
Our experiments for text and images are based on two main codebases. For text experiments, we build upon the [Duo](https://github.com/s-sahoo/duo) codebase. For image experiments, we adapt the [Halton MaskGIT](https://github.com/valeoai/Halton-MaskGIT) codebase. As a result, we maintain separate branches for text and image experiments:
|
| 37 |
|
| 38 |
- Text experiments (besides distillation) are on the `text_pretrain` branch.
|
| 39 |
- Image experiments are on the `image_pretrain` branch.
|
| 40 |
|
| 41 |
-
Additionally, we
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
## Citation
|
| 44 |
```
|
|
|
|
| 16 |
|
| 17 |
[](https://arxiv.org/abs/2506.10892v1)
|
| 18 |
[](hhttps://huggingface.co/jdeschena/pgm)
|
| 19 |
+
[-F9AB00?logo=googlecolab&logoColor=fff)](https://colab.research.google.com/drive/1dhAh4hJ5s89PcQWlE7PoFxF2WBebM6g1)
|
| 20 |
+
[-F9AB00?logo=googlecolab&logoColor=fff)](https://colab.research.google.com/drive/1eRqnK3vasDFqxEq99LqboyoqQyfFI4AC)
|
| 21 |
|
| 22 |
**TL;DR: Partition Generative Models (PGMs) speed up parallel generation by partitioning tokens and using sparse attention instead of masking.**
|
| 23 |
|
|
|
|
| 25 |
<img src="https://jdeschena.github.io/pgm/static/images/pgm_vs_mgm.jpg" width="60%">
|
| 26 |
</div>
|
| 27 |
|
| 28 |
+
## Try Our Models
|
| 29 |
+
Try our models directly on Google Colab!
|
| 30 |
+
- [Image modeling notebook](https://colab.research.google.com/drive/1eRqnK3vasDFqxEq99LqboyoqQyfFI4AC)
|
| 31 |
+
- [Language modeling notebook](https://colab.research.google.com/drive/1dhAh4hJ5s89PcQWlE7PoFxF2WBebM6g1)
|
|
|
|
| 32 |
|
| 33 |
+
## Getting started locally
|
| 34 |
+
To get started, install the dependencies in `requirements.txt`. The requirements *do not* contain the `numpy` and `torch` dependencies, since these need to be set in combination. For us, we work in docker containers, built from `nvcr.io/nvidia/pytorch:25.02-py3`, which uses `torch==2.7.0` and `numpy==1.26.4`.
|
| 35 |
|
| 36 |
+
### Reproducing the Results
|
| 37 |
Our experiments for text and images are based on two main codebases. For text experiments, we build upon the [Duo](https://github.com/s-sahoo/duo) codebase. For image experiments, we adapt the [Halton MaskGIT](https://github.com/valeoai/Halton-MaskGIT) codebase. As a result, we maintain separate branches for text and image experiments:
|
| 38 |
|
| 39 |
- Text experiments (besides distillation) are on the `text_pretrain` branch.
|
| 40 |
- Image experiments are on the `image_pretrain` branch.
|
| 41 |
|
| 42 |
+
Additionally, we distilled models using [SDTT](https://github.com/jdeschena/sdtt). The relevant code can be found on the `text_distill_sdtt` branch, which is a slight adaptation of the [SDTT](https://github.com/jdeschena/sdtt) codebase. You can find further instructions in the respective branches.
|
| 43 |
+
|
| 44 |
+
### Checkpoints
|
| 45 |
+
We release checkpoints trained on OpenWebText (1M steps, distilled and undistilled) and ImageNet (500k steps) on [🤗 Huggingface](https://huggingface.co/jdeschena/pgm). The checkpoints on HuggingFace are directly compatible with the code without conversion.
|
| 46 |
|
| 47 |
## Citation
|
| 48 |
```
|