keras/smolvlm2_500m_video_instruct

Model Overview

Model Overview

SmolVLM2 is a family of lightweight, open multimodal models published by Hugging Face in 256M, 500M, and 2.2B parameter sizes. Each model pairs a SigLIP image encoder with a SmolLM2 text decoder, following the Idefics3 architecture; the 2.2B model uses the shape-optimized SigLIP-SO400M encoder and SmolLM2-1.7B, while the 256M and 500M models use the smaller SigLIP-B/16 encoder with a smaller SmolLM2 decoder. The models take interleaved text, single or multiple images, and video frames as input and generate text, which makes them suitable for image and video captioning, visual question answering, document and chart understanding, and video highlight detection on resource-constrained hardware. SmolVLM2 does not generate images or video, and the publisher states it is not intended for high-stakes decisions affecting a person's well-being or livelihood. See the model card below for benchmarks, data sources, and intended use cases.

Weights are released under the Apache 2.0 License. Keras model code is released under the Apache 2 License.

Links

• [SmolVLM2 Quickstart Notebook](coming soon)
• [SmolVLM2 API Documentation](coming soon)
• SmolVLM2 Model Card
• SmolVLM2 Model Paper
• KerasHub Beginner Guide
• KerasHub Model Publishing Guide

Installation

Keras and KerasHub can be installed with:

pip install -U -q keras-hub
pip install -U -q keras

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the Keras Getting Started page.

Presets

Checkpoints are being uploaded; the table lists the presets defined in KerasHub PR #3096.

The following model checkpoints are provided by the Keras team. Full code examples for each are available below.

Preset name	Parameters	Description
smolvlm2_256m_video_instruct	256.48M	Smallest SmolVLM2 model, instruction-tuned for image and video understanding. Pairs a SigLIP-B/16 vision encoder with a SmolLM2 text decoder.
smolvlm2_500m_video_instruct	507.48M	Mid-size SmolVLM2 model, instruction-tuned for image and video understanding. Pairs a SigLIP-B/16 vision encoder with a SmolLM2 text decoder.
smolvlm2_2.2b_instruct	2.25B	Largest SmolVLM2 model, instruction-tuned for image, multi-image and video understanding. Pairs a SigLIP-SO400M vision encoder with a SmolLM2-1.7B text decoder.

Example Usage

import keras
import keras_hub
import numpy as np

Use generate() to do text generation.

smolvlm2_lm = keras_hub.models.SmolVLM2CausalLM.from_preset("smolvlm2_500m_video_instruct")
smolvlm2_lm.generate("<|im_start|>User:What is Keras?

## Example Usage with Hugging Face URI

```python
import keras
import keras_hub
import numpy as np

Use generate() to do text generation.

smolvlm2_lm = keras_hub.models.SmolVLM2CausalLM.from_preset("hf://keras/smolvlm2_500m_video_instruct")
smolvlm2_lm.generate("<|im_start|>User:What is Keras?