| --- |
| license: mit |
| base_model: zai-org/GLM-Image |
| tags: |
| - core-ai |
| - coreai |
| - text-to-image |
| - autoregressive |
| - diffusion |
| - glm |
| - on-device |
| - apple-silicon |
| pipeline_tag: text-to-image |
| library_name: coreai |
| --- |
| |
| # GLM-Image — Core AI |
|
|
| [ZhipuAI's **GLM-Image**](https://huggingface.co/zai-org/GLM-Image) (16B, MIT) converted to |
| **Core AI** for on-device image generation on Apple Silicon (macOS 27+) — the zoo's first |
| **autoregressive + diffusion hybrid**. |
|
|
| GLM-Image generates in two stages: a **9B GLM-4 autoregressive model** writes the image as a |
| grid of discrete *visual prior tokens* (sampled left-to-right like an LLM, ~36 tok/s on |
| M4 Max), and a **7B flow-matching DiT** denoises the actual pixels conditioned on those |
| tokens, followed by a 16-channel VAE decode. Both native **1024×1024** and a faster |
| **512×512** variant are included. |
|
|
| > **macOS only.** The AR (9.6 GB) + DiT (9.2 GB) weights are resolution-independent, so the |
| > 512 variant is no smaller — an iPhone port needs int4 + sequential component loading. |
|
|
| ## Components |
|
|
| | Component | Description | Size | |
| | --- | --- | --- | |
| | `glm_image_ar.aimodelc` | GLM-4-9B visual-token AR decoder (int8, S=1 decode, AOT h16c GPU) | 9.6 GB | |
| | `glm_image_dit_1024.aimodelc` | 30-block flow-matching DiT, 1024² (int8, AOT h16c GPU) | 9.2 GB | |
| | `glm_image_dit_512.aimodelc` | Same DiT exported at 512² | 9.2 GB | |
| | `glm_image_vae_1024.aimodel` | 16ch AutoencoderKL decoder, 1024² (fp32, CPU) | 0.9 GB | |
| | `glm_image_vae_512.aimodel` | Same VAE exported at 512² | 0.9 GB | |
| | `tokenizer/` | GLM tokenizer (vocab 168 064; visual tokens < 16 512) | — | |
| | `ehs.f32` | Empty-glyph text embedding [1×1×1472] — the DiT text input for glyph-free prompts (no T5 needed at runtime) | 6 KB | |
|
|
| Weights are int8 (per-block-32) for AR/DiT; the VAE stays fp32 (fp16 overflows its |
| activations). Compute precision float16 on GPU. One resolution needs ~19.7 GB on disk |
| (AR + one DiT + one VAE). |
|
|
| ## Usage |
|
|
| ### Sample app (easiest) |
|
|
| [**CoreAIImageGen** (macOS)](https://github.com/john-rocky/coreai-model-zoo/tree/main/apps/CoreAIImageGen) |
| — run the `CoreAIImageGenMac` scheme, pick **GLM-Image 1024 (AR+diffusion)** from the model |
| menu, tap **Download & Load**, type a prompt, **Generate**. |
|
|
| - **Steps 20** (default) ≈ 4.5 min/image on M4 Max — reference quality |
| - **Steps 12** ≈ 2.5 min — nearly indistinguishable |
| - The 512 variant generates in ~1 min |
|
|
| The AR stage is *sampled* (temperature 0.9 / top-p 0.75, the upstream `generation_config`), |
| so re-rolling the seed meaningfully changes composition. |
|
|
| ### Pipeline contract (build your own) |
|
|
| The hybrid does not fit Apple's high-level `CoreAIDiffusionPipeline`; the host loop is |
| ~300 lines of Swift (see the app's `GlmImagePipeline`): |
|
|
| 1. **AR**: `tokenize(prompt) + fixed grid suffix` → S=1 prefill/decode with host-computed |
| 3D-mRoPE cos/sin → sample visual tokens → 2× upsample the large grid → `prior[1, N]` |
| (N = 1024 @512, 4096 @1024). |
| 2. **DiT**: 20-step CFG (guidance 1.5, `prior_scale` 1/0 for cond/uncond). Condition the |
| DiT on the **raw integer schedule** `trunc(linspace(1000,1,steps+1)) − 1`; step the |
| latent with the **mu-shifted sigmas** `σ' = μ/(μ + (1/σ − 1))`, μ = 0.75·side/256 + 0.25. |
| 3. **VAE** (fp32, CPU): latent → RGB. |
|
|
| ⚠️ Feeding the DiT the *shifted* timesteps (sigma×1000) instead of the raw schedule skews |
| the adaLN time conditioning every step and drifts colors — resolution-dependently (mild |
| at 512, strong at 1024). This is the one non-obvious contract in the port. |
|
|
| ## Limitations |
|
|
| - **Glyph rendering (text in images) is not wired**: prompts containing quoted text |
| (`'…'`, `"…"`, `「…」`) route through a T5 glyph encoder that is not part of this bundle. |
| Keep prompts glyph-free. |
| - Negative prompts are a no-op (GLM's CFG drops the prior tokens instead). |
| - Image-to-image / editing (`GlmImageKVCache`) not yet exported. |
|
|
| ## Quality |
|
|
| Verified against the bf16 diffusers reference: DiT single-forward cosine 0.9999, VAE |
| byte-exact, AR prior-injection reproduces the reference composition; end-to-end output is |
| visually on par with `GlmImagePipeline` at the same step count. |
|
|
| ## License |
|
|
| MIT (inherited from [zai-org/GLM-Image](https://huggingface.co/zai-org/GLM-Image)). |
|
|
