| --- |
| tags: |
| - gemma3 |
| - xnnpack |
| --- |
| |
| # Gemma 3 4B-IT (ExecuTorch, XNNPACK, 8da4w) |
|
|
| This folder contains an ExecuTorch .pte export of https://huggingface.co/google/gemma-3-4b-it for CPU inference via the XNNPACK backend, with post-training quantization enabled. Gemma 3 is a multimodal vision-language model |
| supporting both text and image inputs. |
|
|
| Quantization |
|
|
| - --qlinear 8da4w: linear layers (text decoder) use 8-bit dynamic activations + 4-bit weights |
| - --qlinear_encoder 8da4w: linear layers (vision encoder) use 8-bit dynamic activations + 4-bit weights |
| - --qembedding 8w: embeddings use 8-bit weights |
| |
| ## Export |
| ``` |
| optimum-cli export executorch \ |
| --model "google/gemma-3-4b-it" \ |
| --task "multimodal-text-to-text" \ |
| --recipe "xnnpack" \ |
| --use_custom_sdpa \ |
| --use_custom_kv_cache \ |
| --qlinear "8da4w" \ |
| --qlinear_encoder "8da4w" \ |
| --qembedding "8w" \ |
| --output_dir "<output_dir>" |
| ``` |
| |
| ## Run |
| Build the runner from the ExecuTorch repo root |
| ``` |
| make gemma3-cpu |
| ``` |
| Download tokenizer |
| ``` |
| curl -L https://huggingface.co/google/gemma-3-4b-it/resolve/main/tokenizer.json -o tokenizer.json |
| ``` |
| Run model |
| ``` |
| ./cmake-out/examples/models/gemma3/gemma3_e2e_runner \ |
| --model_path "model.pte" \ |
| --tokenizer_path "tokenizer.json" \ |
| --image_path "docs/source/_static/img/et-logo.png" \ |
| --temperature 0 |
| ``` |
