README.md

---
license: apache-2.0
base_model:
- google/siglip-so400m-patch14-384
pipeline_tag: zero-shot-image-classification
tags:
- siglip
- vision-language
- dual-encoder
- renesas
- x5h
---

# SigLIP-SO400M-patch14-384 - Renesas X5H

## Introduction

This repository contains the **SigLIP-SO400M-patch14-384** dual-encoder model, optimized for the **Renesas X5H** platform for **zero-shot image-text similarity** inference.

SigLIP is a dual-encoder model consisting of a vision encoder and a text encoder. It is **not** a generative model -- it does not produce text output. Instead, it computes similarity scores between images and text labels, enabling zero-shot image classification and image-text matching.

- **Model Architecture:** SigLIP uses a Vision Transformer (ViT) for image encoding and a Transformer for text encoding. Both encoders produce L2-normalized embeddings that are compared via dot-product similarity with a learned scale and bias, followed by sigmoid activation.
- **Model Summary:**
  |   Parameter    |    SigLIP-SO400M-patch14-384     |
  |:---:|:---:|
  | HIDDEN_SIZE | 1152 |
  | INTERMEDIATE_SIZE | 4304 |
  | NUM_HEADS | 16 |
  | HEAD_DIM | 72 |
  | NUM_LAYERS | 27 (both encoders) |
  | PATCH_SIZE | 14 |
  | IMAGE_SIZE | 384 |
  | NUM_PATCHES | 729 |
  | VOCAB_SIZE | 32000 |
  | MAX_TEXT_SEQ_LEN | 64 |
- **Source Model:** [google/siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384)



## Performance
The following performance metrics were measured on the Renesas X5H board.

| Precision | Device | VE Latency (ms) | TE Latency (ms) | NPU DDR (MB) |
|:---:|:---:|:---:|:---:|:---:|
| FP16 | NPX6 | 396.0 | 43.0 | 1708.04 |



## Prerequisites
To run the model, you need:

1.  **Renesas X5H Board with SDK v4.32.0 or v4.34.0**
2.  **Hugging Face CLI**: For downloading the model and installer.

## Board Setup

### Power Cycle

The X5H board must be power-cycled via USB serial before first use or after any NPU hang.

```bash
echo "POWER#OF" > /dev/ttyUSB2
sleep 3
echo "POWER#ON" > /dev/ttyUSB2
```

Wait for the board to boot (typically 30-60 seconds).

### NPU Setup

Run `setup_npu.sh` **exactly once** after each power cycle. Do not run it multiple times without rebooting first.

```bash
bash ./setup_npu.sh npu0
```

This loads kernel modules and starts all 14 NPU firmware cores.

**Important**: FP16 and W4A16 runners cannot be used back-to-back on the same boot. Running W4A16 corrupts NPU internal state, causing subsequent FP16 runs to produce NaN. Power cycle the board when switching between variants.

## Deployment

### SigLIP-SO400M-patch14-384 (FP16)

Two versions are available:
- **xOS-v4.34**:  VE encoding 

1. Download the installer **siglip-runner-3.0.0-Linux.sh** from **Files and versions** tab under **fp16/binaries/rcar-x5hv1/xOS-v4.34/** (optimized) or **fp16/binaries/rcar-x5hv1/xOS-v4.32/** (baseline).
2. Copy the installer to the X5H board and run the installer.
   ```bash
   bash ./siglip-runner-3.0.0-Linux.sh --prefix=./ --exclude-subdir --skip-license
   ```
3. Expected directory structure on the X5H board.
   ```bash
    siglip-runner
    ├── model
    │   └── mmproj-siglip-f16.gguf
    ├── firmwares
    ├── kernel_modules
    ├── scripts
    ├── test_data
    │   └── car-1.ppm
    ├── siglip-runner
    └── setup_npu.sh
   ```

### Inference - SigLIP-SO400M-patch14-384 (FP16)
```bash
bash ./setup_npu.sh npu0
./siglip-runner -m model/mmproj-siglip-f16.gguf -i test_data/car-1.ppm -t 262,266,1304,267,262,266,616,1 -s
```

Expected output:
```
similarity_score: 6.5136025660e-03 (0.0065136026)
```



### SigLIP-SO400M-patch14-384 (W4A16)
1. Download the installer **siglip-w4a16-runner-3.0.0-Linux.sh** from **Files and versions** tab under **w4a16/binaries/rcar-x5hv1/xOS-v4.34/** folder.
2. Copy the installer to the X5H board and run the installer.
   ```bash
   bash ./siglip-w4a16-runner-3.0.0-Linux.sh --prefix=./ --exclude-subdir --skip-license
   ```
3. Expected directory structure on the X5H board.
   ```bash
    siglip-w4a16-runner
    ├── model
    │   └── mmproj-siglip-f16.gguf
    ├── graphs
    │   ├── siglip-ve-w4a16
    │   └── siglip-te-w4a16
    ├── firmwares
    ├── kernel_modules
    ├── scripts
    ├── test_data
    │   └── car-1.ppm
    ├── siglip-w4a16-runner
    └── setup_npu.sh
   ```

### Inference - SigLIP-SO400M-patch14-384 (W4A16)
```bash
bash ./setup_npu.sh npu0
./siglip-w4a16-runner -m model/mmproj-siglip-f16.gguf -i test_data/car-1.ppm -t 262,266,1304,267,262,266,616,1 -g graphs -s
```

Expected output:
```
similarity_score: 5.3571168333e-02 (0.0535711683)
```

## CLI Reference

```
Usage: siglip-runner -m <gguf> (-i <image> | -L <image_list>) (-t <token_ids> | -f <prompts_file>) [options]
  -m  Path to SigLIP GGUF model
  -i  Path to image (JPEG, PNG, BMP, or PPM)
  -L  File with one image path per line (batch image mode)
  -t  Comma-separated token IDs (single prompt mode)
  -f  Prompts file for batch mode (label|token_ids per line)
  -d  NPU device path (default: /dev/snps/arcnet0/app0)
  -s  Print performance metrics

W4A16 adds:
  -g  Graph directory (contains siglip-ve-w4a16/ and siglip-te-w4a16/)
```

Token IDs are SentencePiece encoded. The runner pads to 64 tokens internally and
places EOS (token ID 1) at position 63.