granite-vision-4.1-0.6B-small

This is a small version of ibm-granite/granite-vision-4.1-4b created for testing and development. It keeps every hyperparameter of the base model identical (hidden sizes, head counts, vocabulary, muP scalers, projector geometry, image preprocessing) and only reduces the number of text/vision decoder layers, the deepstack/spatial injection-point lists (consistently with the new layer counts), and nothing else.

Model Details

• Base Model: ibm-granite/granite-vision-4.1-4b
• Architecture: Granite4VisionForConditionalGeneration (siglip vision tower + window-QFormer deepstack/spatial projectors + granite text tower; text is dense, so there is no "A" (activated) part in the name)
• Total Parameters: 0.57B

Configuration Changes

Only the allowed change classes were applied; everything else is identical to the base:

Parameter	Base	Small
text_config.num_hidden_layers	40	2
vision_config.num_hidden_layers	27	2
deepstack_layer_map	[[-19, 9], [-13, 6], [-7, 3], [-1, 0]] (4 layerwise projectors)	[[-1, 0]] (1 layerwise projector; verbatim final base entry)
spatial_target_layers	[12, 15, 18, 21]	[1]

The granite text tower is dense with uniform full attention (a single layer type), so 2 layers cover all distinct text layer types; the siglip vision tower likewise has a single layer type (2 layers). The injection-point lists were trimmed consistently with the new layer counts so that both distinguishing mechanisms of this architecture remain exercised: the deepstack injection (vision hidden state [-1] -> layerwise projector -> text layer 0) and the spatial-sampling injection (vision hidden state [-1] (=spatial_vision_layer) -> spatial projector group 0 -> text layer 1). The qformer inside every projector keeps its base 1 encoder layer (already minimal). All 4 spatial projector modules exist as in the base (the modeling hardcodes 4 offset groups); with spatial_target_layers=[1] only group 0 receives features on text layer 1. All other values are unchanged, including hidden_size 2560 (text) / 1152 (vision+qformer), vocab_size 100352, tied embeddings, 40/8 attention heads, downsample_rate "4/8", image_grid_pinpoints, muP scalers (embedding_multiplier 12.0, logits_scaling 10.0, residual_multiplier 0.22, attention_multiplier 0.015625), and rope_theta 1e7.

Checkpoint Structure

Single-file bf16 model.safetensors (239 tensors) using the original base checkpoint naming: model.language_model.* (granite, tied embeddings - no lm_head), model.vision_tower.vision_model.* (siglip, wrapped naming as in the base; transformers 5.17 flattens SiglipVisionModel internally, so the saved names were converted back by reversing the PrefixChange(prefix_to_remove="vision_model") mapping), model.layerwise_projectors.0.* and model.spatial_projectors.{0..3}.* (window-QFormer downsamplers with blip2-style qformers), and model.image_newline. Tensor-name patterns match the base model.safetensors.index.json exactly, modulo layer/projector counts (validated programmatically: 0 extra patterns; all per-component key sets identical to the base's). The checkpoint loads through AutoModelForImageTextToText.from_pretrained with the same conversion path as the base repo, and all conversions round-trip bit-exactly.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained("inference-optimization/granite-vision-4.1-0.6B-small", device_map="auto")
processor = AutoProcessor.from_pretrained("inference-optimization/granite-vision-4.1-0.6B-small")

input_ids = processor.tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(processor.tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

1. Loaded the base AutoConfig and applied only the allowed changes (2 text layers, 2 vision layers, trimmed injection-point lists).
2. Built the model with skip_weights_download, re-initialized weights (init_weights() + NaN fixup), cast to bf16, and saved.
3. Converted the saved checkpoint to the base naming (re-adding the vision_model. prefix in the siglip tower).
4. Fine-tuned on the skill's text-only copypasta dataset (loaded with AutoModelForImageTextToText, text-only forward) in bf16 until target perplexity was reached.
5. Re-converted the fine-tuned checkpoint to base naming and validated structure + round-trip against the base index.

Validation

• Perplexity on the skill's validation text: 2.73 (target <= 10)
• Greedy generation from "According to all known laws": produces the trained copypasta continuation correctly.
• Multimodal smoke test: a full image+text forward through the vision tower, deepstack and spatial projectors produces finite logits and generates (the model was only trained on text, so image content is not meaningful).

Notes

• Corrected base facts: the text tower of the base is granite with hidden_size 2560 and 40 dense, uniform full-attention layers (no sliding-window/mamba hybrid fields exist in this checkpoint's config); the 1152-hidden/27-layer numbers belong to the siglip vision tower. With hidden 2560 and vocab 100352, the tied embedding alone is 0.26B, so ~0.57B is close to the practical floor for this architecture (the 4 spatial projector modules are hardcoded by the modeling and add ~83M).
• The base repo ships remote-code files (configuration.py, modeling.py, ...) and carries auto_map entries; transformers 5.17 has native granite4_vision support which this model uses, so auto_map was removed from config.json/processor configs and the remote-code files are not included (same convention as the DeepSeek-V3 small model).
• Fine-tuning note: like other muP-scaled models, the first training run (lr 5e-5) plateaued at high loss because the logits are divided by logits_scaling (10.0); continuing from that checkpoint with lr 2e-4 converged to the target perplexity.
• No quantization_config in the base; nothing stripped. bf16 throughout.