Update README.md
Browse files
README.md
CHANGED
|
@@ -1,148 +1,148 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
pipeline_tag: text-to-image
|
| 4 |
-
library_name: diffusers
|
| 5 |
-
tags:
|
| 6 |
-
- text-to-image
|
| 7 |
-
- quantized
|
| 8 |
-
- int8
|
| 9 |
-
- ernie-image
|
| 10 |
-
- diffusers
|
| 11 |
-
base_model:
|
| 12 |
-
base_model_relation: quantized
|
| 13 |
-
---
|
| 14 |
-
|
| 15 |
-
# ERNIE-Image-INT8
|
| 16 |
-
|
| 17 |
-
ERNIE-Image-INT8 is a publishable INT8 derivative of `Baidu/ERNIE-Image`, prepared for local deployment, packaging, and reproducible benchmarking. The default release profile prioritizes `transformer` INT8 quantization first, while `text_encoder` and `pe` may remain in `bfloat16` when quality checks show that full INT8 introduces unacceptable degradation.
|
| 18 |
-
|
| 19 |
-
## What Is Included
|
| 20 |
-
|
| 21 |
-
- Diffusers-compatible model folder layout.
|
| 22 |
-
- Component-wise precision manifest and quantization metadata.
|
| 23 |
-
|
| 24 |
-
## Precision Matrix
|
| 25 |
-
|
| 26 |
-
| Component | Backend | Precision | Enabled |
|
| 27 |
-
| --- | --- | --- | --- |
|
| 28 |
-
| transformer | quanto | int8 | True |
|
| 29 |
-
| text_encoder | none | bfloat16 | False |
|
| 30 |
-
| pe | none | bfloat16 | False |
|
| 31 |
-
|
| 32 |
-
## Recommended Runtime
|
| 33 |
-
|
| 34 |
-
- NVIDIA GPU with 24 GB+ VRAM for practical generation.
|
| 35 |
-
- CPU is supported only for loading validation, metadata inspection, and smoke tests.
|
| 36 |
-
- Recommended image sizes follow the original ERNIE-Image guidance: `1024x1024`, `848x1264`, `1264x848`, `1200x896`.
|
| 37 |
-
|
| 38 |
-
## Quick Start
|
| 39 |
-
|
| 40 |
-
```python
|
| 41 |
-
import torch
|
| 42 |
-
from diffusers import ErnieImagePipeline
|
| 43 |
-
|
| 44 |
-
pipe = ErnieImagePipeline.from_pretrained(
|
| 45 |
-
"ixim/ERNIE-Image-INT8",
|
| 46 |
-
torch_dtype=torch.bfloat16,
|
| 47 |
-
).to("cuda")
|
| 48 |
-
|
| 49 |
-
image = pipe(
|
| 50 |
-
prompt="A premium event poster with readable bilingual typography and strong layout hierarchy.",
|
| 51 |
-
width=848,
|
| 52 |
-
height=1264,
|
| 53 |
-
num_inference_steps=50,
|
| 54 |
-
guidance_scale=4.0,
|
| 55 |
-
use_pe=True,
|
| 56 |
-
).images[0]
|
| 57 |
-
|
| 58 |
-
image.save("output.png")
|
| 59 |
-
```
|
| 60 |
-
|
| 61 |
-
## Benchmark Snapshot
|
| 62 |
-
|
| 63 |
-
Benchmark context: 7 prompt(s), seed=42, prompts=zh_portrait_studio_east_asian, zh_poster_dense_text, zh_infographic_wide, zh_browser_ui_article, en_storyboard_dialogue, zh_sticker_grid, en_backlit_street_photo. Primary comparison covers transformer-int8 + pe-bf16 + use_pe=true, transformer-int8 + pe-int8 + use_pe=true, transformer-int8 + use_pe=false; variant-specific steps / guidance_scale / use_pe are listed in the tables below. Supplementary references cover ERNIE-Image-Turbo Reference. The `pe-int8` row is a runtime-quantized benchmark variant used for comparison only, and does not change the packaged release precision matrix shown above. Peak VRAM reports the peak reserved CUDA memory of the current PyTorch process during each generation call.
|
| 64 |
-
|
| 65 |
-
| Group | Variant | Prompt Count | Avg Latency (ms) | Avg Peak VRAM (MiB) | Steps | CFG | Use PE |
|
| 66 |
-
| --- | --- | --- | --- | --- | --- | --- | --- |
|
| 67 |
-
| primary | transformer-int8 + pe-bf16 + use_pe=true | 7 | 78053 | 28516 | 50 | 4.0 | True |
|
| 68 |
-
| primary | transformer-int8 + pe-int8 + use_pe=true | 7 | 81412 | 28721 | 50 | 4.0 | True |
|
| 69 |
-
| primary | transformer-int8 + use_pe=false | 7 | 60287 | 28339 | 50 | 4.0 | False |
|
| 70 |
-
| supplementary | ERNIE-Image-Turbo Reference | 7 | 32535 | 35255 | 8 | 1.0 | True |
|
| 71 |
-
|
| 72 |
-
### Prompt-by-Prompt Comparison
|
| 73 |
-
|
| 74 |
-
#### zh_portrait_studio_east_asian
|
| 75 |
-
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 76 |
-
| --- | --- | --- | --- | --- | --- |
|
| 77 |
-
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 70842 ms / 28968 MiB peak |
|
| 78 |
-
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 76522 ms / 29172 MiB peak |
|
| 79 |
-
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 60728 ms / 28790 MiB peak |
|
| 80 |
-
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 20393 ms / 35708 MiB peak |
|
| 81 |
-
|
| 82 |
-
#### zh_poster_dense_text
|
| 83 |
-
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 84 |
-
| --- | --- | --- | --- | --- | --- |
|
| 85 |
-
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 71753 ms / 27912 MiB peak |
|
| 86 |
-
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 73674 ms / 28118 MiB peak |
|
| 87 |
-
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 60753 ms / 27738 MiB peak |
|
| 88 |
-
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 22736 ms / 34654 MiB peak |
|
| 89 |
-
|
| 90 |
-
#### zh_infographic_wide
|
| 91 |
-
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 92 |
-
| --- | --- | --- | --- | --- | --- |
|
| 93 |
-
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 72318 ms / 27914 MiB peak |
|
| 94 |
-
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 80810 ms / 28120 MiB peak |
|
| 95 |
-
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 60691 ms / 27738 MiB peak |
|
| 96 |
-
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 22987 ms / 34654 MiB peak |
|
| 97 |
-
|
| 98 |
-
#### zh_browser_ui_article
|
| 99 |
-
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 100 |
-
| --- | --- | --- | --- | --- | --- |
|
| 101 |
-
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 74728 ms / 27916 MiB peak |
|
| 102 |
-
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 76007 ms / 28120 MiB peak |
|
| 103 |
-
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 61152 ms / 27738 MiB peak |
|
| 104 |
-
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 21412 ms / 34654 MiB peak |
|
| 105 |
-
|
| 106 |
-
#### en_storyboard_dialogue
|
| 107 |
-
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 108 |
-
| --- | --- | --- | --- | --- | --- |
|
| 109 |
-
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 93052 ms / 28968 MiB peak |
|
| 110 |
-
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 89542 ms / 29172 MiB peak |
|
| 111 |
-
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 59642 ms / 28790 MiB peak |
|
| 112 |
-
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 55379 ms / 35706 MiB peak |
|
| 113 |
-
|
| 114 |
-
#### zh_sticker_grid
|
| 115 |
-
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 116 |
-
| --- | --- | --- | --- | --- | --- |
|
| 117 |
-
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 92627 ms / 28968 MiB peak |
|
| 118 |
-
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 99956 ms / 29172 MiB peak |
|
| 119 |
-
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 59401 ms / 28790 MiB peak |
|
| 120 |
-
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 62985 ms / 35706 MiB peak |
|
| 121 |
-
|
| 122 |
-
#### en_backlit_street_photo
|
| 123 |
-
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 124 |
-
| --- | --- | --- | --- | --- | --- |
|
| 125 |
-
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 71049 ms / 28968 MiB peak |
|
| 126 |
-
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 73374 ms / 29172 MiB peak |
|
| 127 |
-
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 59643 ms / 28790 MiB peak |
|
| 128 |
-
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 21852 ms / 35706 MiB peak |
|
| 129 |
-
|
| 130 |
-
## Example Prompt Set
|
| 131 |
-
|
| 132 |
-
See `example_prompts.json` for the curated prompt suite used during packaging and regression checks. When `scripts/build_release.py` receives an `--examples-dir` benchmark folder, the prompt-grouped benchmark tables above also render preview images from those outputs automatically.
|
| 133 |
-
|
| 134 |
-
## Intended Use
|
| 135 |
-
|
| 136 |
-
- Local image generation tools and controlled packaging workflows.
|
| 137 |
-
- Quantization research on large open-weight text-to-image models.
|
| 138 |
-
- Internal demo services where image history, prompt reproducibility, and artifact packaging matter.
|
| 139 |
-
|
| 140 |
-
## Limitations
|
| 141 |
-
|
| 142 |
-
- Full CPU generation is not a practical primary target for this release.
|
| 143 |
-
- Text rendering, dense layouts, and long structured prompts should always be rechecked after quantization.
|
| 144 |
-
- Experimental all-INT8 variants can degrade typography, object counting, and layout adherence.
|
| 145 |
-
|
| 146 |
-
## License
|
| 147 |
-
|
| 148 |
-
This release inherits the Apache-2.0 terms of the base model. Review the included `LICENSE` and make sure your downstream usage also complies with the original ERNIE-Image terms and any third-party dependencies you add around it.
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
pipeline_tag: text-to-image
|
| 4 |
+
library_name: diffusers
|
| 5 |
+
tags:
|
| 6 |
+
- text-to-image
|
| 7 |
+
- quantized
|
| 8 |
+
- int8
|
| 9 |
+
- ernie-image
|
| 10 |
+
- diffusers
|
| 11 |
+
base_model: baidu/ERNIE-Image
|
| 12 |
+
base_model_relation: quantized
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# ERNIE-Image-INT8
|
| 16 |
+
|
| 17 |
+
ERNIE-Image-INT8 is a publishable INT8 derivative of `Baidu/ERNIE-Image`, prepared for local deployment, packaging, and reproducible benchmarking. The default release profile prioritizes `transformer` INT8 quantization first, while `text_encoder` and `pe` may remain in `bfloat16` when quality checks show that full INT8 introduces unacceptable degradation.
|
| 18 |
+
|
| 19 |
+
## What Is Included
|
| 20 |
+
|
| 21 |
+
- Diffusers-compatible model folder layout.
|
| 22 |
+
- Component-wise precision manifest and quantization metadata.
|
| 23 |
+
|
| 24 |
+
## Precision Matrix
|
| 25 |
+
|
| 26 |
+
| Component | Backend | Precision | Enabled |
|
| 27 |
+
| --- | --- | --- | --- |
|
| 28 |
+
| transformer | quanto | int8 | True |
|
| 29 |
+
| text_encoder | none | bfloat16 | False |
|
| 30 |
+
| pe | none | bfloat16 | False |
|
| 31 |
+
|
| 32 |
+
## Recommended Runtime
|
| 33 |
+
|
| 34 |
+
- NVIDIA GPU with 24 GB+ VRAM for practical generation.
|
| 35 |
+
- CPU is supported only for loading validation, metadata inspection, and smoke tests.
|
| 36 |
+
- Recommended image sizes follow the original ERNIE-Image guidance: `1024x1024`, `848x1264`, `1264x848`, `1200x896`.
|
| 37 |
+
|
| 38 |
+
## Quick Start
|
| 39 |
+
|
| 40 |
+
```python
|
| 41 |
+
import torch
|
| 42 |
+
from diffusers import ErnieImagePipeline
|
| 43 |
+
|
| 44 |
+
pipe = ErnieImagePipeline.from_pretrained(
|
| 45 |
+
"ixim/ERNIE-Image-INT8",
|
| 46 |
+
torch_dtype=torch.bfloat16,
|
| 47 |
+
).to("cuda")
|
| 48 |
+
|
| 49 |
+
image = pipe(
|
| 50 |
+
prompt="A premium event poster with readable bilingual typography and strong layout hierarchy.",
|
| 51 |
+
width=848,
|
| 52 |
+
height=1264,
|
| 53 |
+
num_inference_steps=50,
|
| 54 |
+
guidance_scale=4.0,
|
| 55 |
+
use_pe=True,
|
| 56 |
+
).images[0]
|
| 57 |
+
|
| 58 |
+
image.save("output.png")
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
## Benchmark Snapshot
|
| 62 |
+
|
| 63 |
+
Benchmark context: 7 prompt(s), seed=42, prompts=zh_portrait_studio_east_asian, zh_poster_dense_text, zh_infographic_wide, zh_browser_ui_article, en_storyboard_dialogue, zh_sticker_grid, en_backlit_street_photo. Primary comparison covers transformer-int8 + pe-bf16 + use_pe=true, transformer-int8 + pe-int8 + use_pe=true, transformer-int8 + use_pe=false; variant-specific steps / guidance_scale / use_pe are listed in the tables below. Supplementary references cover ERNIE-Image-Turbo Reference. The `pe-int8` row is a runtime-quantized benchmark variant used for comparison only, and does not change the packaged release precision matrix shown above. Peak VRAM reports the peak reserved CUDA memory of the current PyTorch process during each generation call.
|
| 64 |
+
|
| 65 |
+
| Group | Variant | Prompt Count | Avg Latency (ms) | Avg Peak VRAM (MiB) | Steps | CFG | Use PE |
|
| 66 |
+
| --- | --- | --- | --- | --- | --- | --- | --- |
|
| 67 |
+
| primary | transformer-int8 + pe-bf16 + use_pe=true | 7 | 78053 | 28516 | 50 | 4.0 | True |
|
| 68 |
+
| primary | transformer-int8 + pe-int8 + use_pe=true | 7 | 81412 | 28721 | 50 | 4.0 | True |
|
| 69 |
+
| primary | transformer-int8 + use_pe=false | 7 | 60287 | 28339 | 50 | 4.0 | False |
|
| 70 |
+
| supplementary | ERNIE-Image-Turbo Reference | 7 | 32535 | 35255 | 8 | 1.0 | True |
|
| 71 |
+
|
| 72 |
+
### Prompt-by-Prompt Comparison
|
| 73 |
+
|
| 74 |
+
#### zh_portrait_studio_east_asian
|
| 75 |
+
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 76 |
+
| --- | --- | --- | --- | --- | --- |
|
| 77 |
+
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 70842 ms / 28968 MiB peak |
|
| 78 |
+
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 76522 ms / 29172 MiB peak |
|
| 79 |
+
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 60728 ms / 28790 MiB peak |
|
| 80 |
+
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 20393 ms / 35708 MiB peak |
|
| 81 |
+
|
| 82 |
+
#### zh_poster_dense_text
|
| 83 |
+
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 84 |
+
| --- | --- | --- | --- | --- | --- |
|
| 85 |
+
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 71753 ms / 27912 MiB peak |
|
| 86 |
+
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 73674 ms / 28118 MiB peak |
|
| 87 |
+
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 60753 ms / 27738 MiB peak |
|
| 88 |
+
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 22736 ms / 34654 MiB peak |
|
| 89 |
+
|
| 90 |
+
#### zh_infographic_wide
|
| 91 |
+
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 92 |
+
| --- | --- | --- | --- | --- | --- |
|
| 93 |
+
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 72318 ms / 27914 MiB peak |
|
| 94 |
+
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 80810 ms / 28120 MiB peak |
|
| 95 |
+
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 60691 ms / 27738 MiB peak |
|
| 96 |
+
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 22987 ms / 34654 MiB peak |
|
| 97 |
+
|
| 98 |
+
#### zh_browser_ui_article
|
| 99 |
+
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 100 |
+
| --- | --- | --- | --- | --- | --- |
|
| 101 |
+
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 74728 ms / 27916 MiB peak |
|
| 102 |
+
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 76007 ms / 28120 MiB peak |
|
| 103 |
+
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 61152 ms / 27738 MiB peak |
|
| 104 |
+
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 21412 ms / 34654 MiB peak |
|
| 105 |
+
|
| 106 |
+
#### en_storyboard_dialogue
|
| 107 |
+
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 108 |
+
| --- | --- | --- | --- | --- | --- |
|
| 109 |
+
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 93052 ms / 28968 MiB peak |
|
| 110 |
+
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 89542 ms / 29172 MiB peak |
|
| 111 |
+
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 59642 ms / 28790 MiB peak |
|
| 112 |
+
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 55379 ms / 35706 MiB peak |
|
| 113 |
+
|
| 114 |
+
#### zh_sticker_grid
|
| 115 |
+
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 116 |
+
| --- | --- | --- | --- | --- | --- |
|
| 117 |
+
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 92627 ms / 28968 MiB peak |
|
| 118 |
+
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 99956 ms / 29172 MiB peak |
|
| 119 |
+
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 59401 ms / 28790 MiB peak |
|
| 120 |
+
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 62985 ms / 35706 MiB peak |
|
| 121 |
+
|
| 122 |
+
#### en_backlit_street_photo
|
| 123 |
+
| Variant | Preview | Steps | CFG | Use PE | Latency / Peak VRAM |
|
| 124 |
+
| --- | --- | --- | --- | --- | --- |
|
| 125 |
+
| transformer-int8 + pe-bf16 + use_pe=true |  | 50 | 4.0 | True | 71049 ms / 28968 MiB peak |
|
| 126 |
+
| transformer-int8 + pe-int8 + use_pe=true |  | 50 | 4.0 | True | 73374 ms / 29172 MiB peak |
|
| 127 |
+
| transformer-int8 + use_pe=false |  | 50 | 4.0 | False | 59643 ms / 28790 MiB peak |
|
| 128 |
+
| ERNIE-Image-Turbo Reference |  | 8 | 1.0 | True | 21852 ms / 35706 MiB peak |
|
| 129 |
+
|
| 130 |
+
## Example Prompt Set
|
| 131 |
+
|
| 132 |
+
See `example_prompts.json` for the curated prompt suite used during packaging and regression checks. When `scripts/build_release.py` receives an `--examples-dir` benchmark folder, the prompt-grouped benchmark tables above also render preview images from those outputs automatically.
|
| 133 |
+
|
| 134 |
+
## Intended Use
|
| 135 |
+
|
| 136 |
+
- Local image generation tools and controlled packaging workflows.
|
| 137 |
+
- Quantization research on large open-weight text-to-image models.
|
| 138 |
+
- Internal demo services where image history, prompt reproducibility, and artifact packaging matter.
|
| 139 |
+
|
| 140 |
+
## Limitations
|
| 141 |
+
|
| 142 |
+
- Full CPU generation is not a practical primary target for this release.
|
| 143 |
+
- Text rendering, dense layouts, and long structured prompts should always be rechecked after quantization.
|
| 144 |
+
- Experimental all-INT8 variants can degrade typography, object counting, and layout adherence.
|
| 145 |
+
|
| 146 |
+
## License
|
| 147 |
+
|
| 148 |
+
This release inherits the Apache-2.0 terms of the base model. Review the included `LICENSE` and make sure your downstream usage also complies with the original ERNIE-Image terms and any third-party dependencies you add around it.
|