---
license: gemma
tags:
- image-feature-extraction
- siglip
base_model: google/gemma-3-4b-pt
library_name: transformers
---

# Gemma 3 Vision Encoder (extracted)

This repository contains the SigLIP-family vision encoder extracted from **google/gemma-3-4b-pt**.
It also includes the Gemma multimodal projector weights (state_dict) and a small metadata file.

## Contents

- `config.json`, `model.safetensors`: the SigLIP vision encoder
- `preprocessor_config.json`: the image processor settings used by Gemma 3
- `projector_state_dict.pt`: PyTorch state dict for the Gemma projector
- `projector_config.json`: metadata (class, dims, token count if detected)
- `NOTICE`: Gemma Terms pointer

## Basic usage (encoder as feature extractor)

```python
from transformers import SiglipVisionModel, AutoImageProcessor
from PIL import Image
import torch

repo_id = "<your-username>/<your-repo>"
encoder = SiglipVisionModel.from_pretrained(repo_id).eval()
processor = AutoImageProcessor.from_pretrained(repo_id)

img = Image.open("test.jpg").convert("RGB")
inputs = processor(images=img, return_tensors='pt')
with torch.no_grad():
    feats = encoder(**inputs).last_hidden_state  # (B, Tv, Dv)
print(feats.shape)
```

## Using the projector (Gemma-style multimodal path)

The projector here is provided as a **state dict** plus metadata. It is intended for users
who are wiring a Gemma-style VLM, where the projector maps the vision sequence to a fixed number
of image tokens at the LLM hidden size.

Two common paths:

1) **Use with Transformers' Gemma 3 model**: load the full VLM, then load this projector's state_dict
   into the model's `multi_modal_projector` module.

```python
import torch
from transformers import Gemma3ForConditionalGeneration

repo_id = "<your-username>/<your-repo>"
vlm = Gemma3ForConditionalGeneration.from_pretrained('google/gemma-3-4b-pt', device_map='cpu')
sd = torch.load('projector_state_dict.pt', map_location='cpu')  # or from the repo checkout
vlm.multi_modal_projector.load_state_dict(sd, strict=False)
vlm.eval()
```

2) **Recreate the projector module from the class name**, instantiate it, and load the state dict.
   The metadata file records the fully qualified class name (FQN).

```python
import importlib, json, torch

with open("projector_config.json", "r") as f:
    meta = json.load(f)
fqn = meta.get('projector_fqn')  # e.g., 'transformers.models.gemma3.modeling_gemma3.Gemma3VisionProjector'
mod_name, cls_name = fqn.rsplit('.', 1)
cls = getattr(importlib.import_module(mod_name), cls_name)
projector = cls(**{k: v for k, v in meta.items() if k.endswith('_dim') or k.endswith('_tokens')})
sd = torch.load('projector_state_dict.pt', map_location='cpu')
projector.load_state_dict(sd, strict=False)
projector.eval()
```

## Shapes (for reference)
- Vision hidden size Dv: 1152
- Projector output tokens Ti: 256
- Projector class: `transformers.models.gemma3.modeling_gemma3.Gemma3MultiModalProjector`

## License / Terms

See the `NOTICE` file. Gemma is provided under and subject to the Gemma Terms of Use:
https://ai.google.dev/gemma/terms