File size: 4,926 Bytes
9b8cc2b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | ---
pipeline_tag: image-text-to-text
license: gemma
base_model: google/gemma-3n-E4B
library_name: kerasformers
extra_gated_heading: Access Gemma on Hugging Face
extra_gated_prompt: >-
To access Gemma on Hugging Face, you're required to review and agree to
Google's usage license. To do this, please ensure you're logged in to Hugging
Face and click below. Requests are processed immediately.
extra_gated_button_content: Acknowledge license
license_link: https://ai.google.dev/gemma/terms
language:
- en
tags:
- keras
- kerasformers
- gemma3n
- gemma-3n
- image-text-to-text
- audio-text-to-text
- multimodal
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/kerasformers/gemma-3n-6a7a507adf78dde12680accf) for all versions of Gemma 3n.***
# Run Gemma 3n with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/KerasFormers) [](https://imvision12.github.io/KerasFormers/gemma3n/) [](https://huggingface.co/collections/kerasformers/gemma-3n-6a7a507adf78dde12680accf)
# kerasformers/gemma-3n-e4b
Gemma 3n is Google's on-device **multimodal** (image + audio + text) model. Its
decoder layers several on-device innovations on the Gemma shape: **AltUp**
(alternating updates over parallel hidden streams), **LAuReL** (learned augmented
residuals), **MatFormer** (nested per-layer widths), **Per-Layer Embeddings**, and
activation sparsity, with tail **KV-sharing** and a 5:1 sliding/global attention
schedule. Vision is a **MobileNet-V5** encoder and audio a **USM** conformer, both
feeding soft tokens into the decoder. Base checkpoints are for completion; `-it`
variants are instruction-tuned.
For more details, see Google's original [model card](https://huggingface.co/google/gemma-3n-E4B).
Pure-**Keras 3** conversion of [`google/gemma-3n-E4B`](https://huggingface.co/google/gemma-3n-E4B) for
[kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on
**TensorFlow / Torch / JAX**.
This is a **base** (pretrained) checkpoint, for completion / fine-tuning.
## ✨ Quick start
### Text-only
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from kerasformers.models.gemma3n import Gemma3nTextGenerate, Gemma3nTokenizer
model = Gemma3nTextGenerate.from_weights("kerasformers/gemma-3n-e4b")
tokenizer = Gemma3nTokenizer.from_weights("kerasformers/gemma-3n-e4b")
inputs = tokenizer([{"role": "user", "content": "Hello, who are you?"}])
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0]))
```
### Image + text
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from PIL import Image
from kerasformers.models.gemma3n import Gemma3nConditionalGenerate, Gemma3nProcessor
model = Gemma3nConditionalGenerate.from_weights("kerasformers/gemma-3n-e4b")
processor = Gemma3nProcessor.from_weights("kerasformers/gemma-3n-e4b")
inputs = processor(conversation=[
{"role": "user", "content": [
{"type": "image", "image": Image.open("cat.jpg")},
{"type": "text", "text": "Describe this image in one sentence."},
]}
])
outputs = model.generate(**inputs, max_new_tokens=64)
print(processor.decode(outputs[0]))
```
All Gemma 3n variants load the same way with `from_weights("kerasformers/<variant>")`:
| Variant | Hub | Type |
|---|---|---|
| `gemma-3n-e2b` | [`kerasformers/gemma-3n-e2b`](https://huggingface.co/kerasformers/gemma-3n-e2b) | multimodal / base |
| `gemma-3n-e2b-it` | [`kerasformers/gemma-3n-e2b-it`](https://huggingface.co/kerasformers/gemma-3n-e2b-it) | multimodal / instruct |
| `gemma-3n-e4b` | [`kerasformers/gemma-3n-e4b`](https://huggingface.co/kerasformers/gemma-3n-e4b) | multimodal / base |
| `gemma-3n-e4b-it` | [`kerasformers/gemma-3n-e4b-it`](https://huggingface.co/kerasformers/gemma-3n-e4b-it) | multimodal / instruct |
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
- Loads in **bfloat16** by default (the weights are bf16). Pass
`load_dtype="float32"` for full precision, or `quantization="int8"` to shrink further.
- Gemma 3n is **audio-capable** too: pass `audio` content items in the
conversation to transcribe / reason over speech.
- See [Gemma 3n docs](https://imvision12.github.io/KerasFormers/gemma3n/) and
[Loading Weights](https://imvision12.github.io/KerasFormers/loading_weights/).
- Upstream safetensors still work via the `hf:` prefix, e.g.
`Gemma3nConditionalGenerate.from_weights("hf:google/gemma-3n-E4B")`.
## Special Thanks
A huge thank you to the Google Gemma authors for creating and releasing these models.
License: Gemma (gated). Accept the license on the upstream Hub card before downloading.
|