SigLIP-SO400M-patch14-384 - Renesas X5H

Introduction

This repository contains the SigLIP-SO400M-patch14-384 dual-encoder model, optimized for the Renesas X5H platform for zero-shot image-text similarity inference.

SigLIP is a dual-encoder model consisting of a vision encoder and a text encoder. It is not a generative model -- it does not produce text output. Instead, it computes similarity scores between images and text labels, enabling zero-shot image classification and image-text matching.

• Model Architecture: SigLIP uses a Vision Transformer (ViT) for image encoding and a Transformer for text encoding. Both encoders produce L2-normalized embeddings that are compared via dot-product similarity with a learned scale and bias, followed by sigmoid activation.
• Model Summary:

  Parameter	SigLIP-SO400M-patch14-384
  HIDDEN_SIZE	1152
  INTERMEDIATE_SIZE	4304
  NUM_HEADS	16
  HEAD_DIM	72
  NUM_LAYERS	27 (both encoders)
  PATCH_SIZE	14
  IMAGE_SIZE	384
  NUM_PATCHES	729
  VOCAB_SIZE	32000
  MAX_TEXT_SEQ_LEN	64

• Source Model: google/siglip-so400m-patch14-384

Performance

The following performance metrics were measured on the Renesas X5H board.

Precision	Device	VE Latency (ms)	TE Latency (ms)	NPU DDR (MB)
FP16	NPX6	396.0	43.0	1708.04

Prerequisites

To run the model, you need:

1. Renesas X5H Board with SDK v4.32.0 or v4.34.0
2. Hugging Face CLI: For downloading the model and installer.

Board Setup

Power Cycle

The X5H board must be power-cycled via USB serial before first use or after any NPU hang.

echo "POWER#OF" > /dev/ttyUSB2
sleep 3
echo "POWER#ON" > /dev/ttyUSB2

Wait for the board to boot (typically 30-60 seconds).

NPU Setup

Run setup_npu.sh exactly once after each power cycle. Do not run it multiple times without rebooting first.

bash ./setup_npu.sh npu0

This loads kernel modules and starts all 14 NPU firmware cores.

Important: FP16 and W4A16 runners cannot be used back-to-back on the same boot. Running W4A16 corrupts NPU internal state, causing subsequent FP16 runs to produce NaN. Power cycle the board when switching between variants.

Deployment

SigLIP-SO400M-patch14-384 (FP16)

Two versions are available:

• xOS-v4.34: VE encoding

1. Download the installer siglip-runner-3.0.0-Linux.sh from Files and versions tab under fp16/binaries/rcar-x5hv1/xOS-v4.34/ (optimized) or fp16/binaries/rcar-x5hv1/xOS-v4.32/ (baseline).
2. Copy the installer to the X5H board and run the installer.

   bash ./siglip-runner-3.0.0-Linux.sh --prefix=./ --exclude-subdir --skip-license
   

3. Expected directory structure on the X5H board.

    siglip-runner
    ├── model
    │   └── mmproj-siglip-f16.gguf
    ├── firmwares
    ├── kernel_modules
    ├── scripts
    ├── test_data
    │   └── car-1.ppm
    ├── siglip-runner
    └── setup_npu.sh
   

Inference - SigLIP-SO400M-patch14-384 (FP16)

bash ./setup_npu.sh npu0
./siglip-runner -m model/mmproj-siglip-f16.gguf -i test_data/car-1.ppm -t 262,266,1304,267,262,266,616,1 -s

Expected output:

similarity_score: 6.5136025660e-03 (0.0065136026)

SigLIP-SO400M-patch14-384 (W4A16)

1. Download the installer siglip-w4a16-runner-3.0.0-Linux.sh from Files and versions tab under w4a16/binaries/rcar-x5hv1/xOS-v4.34/ folder.
2. Copy the installer to the X5H board and run the installer.

   bash ./siglip-w4a16-runner-3.0.0-Linux.sh --prefix=./ --exclude-subdir --skip-license
   

3. Expected directory structure on the X5H board.

    siglip-w4a16-runner
    ├── model
    │   └── mmproj-siglip-f16.gguf
    ├── graphs
    │   ├── siglip-ve-w4a16
    │   └── siglip-te-w4a16
    ├── firmwares
    ├── kernel_modules
    ├── scripts
    ├── test_data
    │   └── car-1.ppm
    ├── siglip-w4a16-runner
    └── setup_npu.sh
   

Inference - SigLIP-SO400M-patch14-384 (W4A16)

bash ./setup_npu.sh npu0
./siglip-w4a16-runner -m model/mmproj-siglip-f16.gguf -i test_data/car-1.ppm -t 262,266,1304,267,262,266,616,1 -g graphs -s

Expected output:

similarity_score: 5.3571168333e-02 (0.0535711683)

CLI Reference

Usage: siglip-runner -m <gguf> (-i <image> | -L <image_list>) (-t <token_ids> | -f <prompts_file>) [options]
  -m  Path to SigLIP GGUF model
  -i  Path to image (JPEG, PNG, BMP, or PPM)
  -L  File with one image path per line (batch image mode)
  -t  Comma-separated token IDs (single prompt mode)
  -f  Prompts file for batch mode (label|token_ids per line)
  -d  NPU device path (default: /dev/snps/arcnet0/app0)
  -s  Print performance metrics

W4A16 adds:
  -g  Graph directory (contains siglip-ve-w4a16/ and siglip-te-w4a16/)

Token IDs are SentencePiece encoded. The runner pads to 64 tokens internally and places EOS (token ID 1) at position 63.