Inkling-2-Small-Vision

This repository packages the vision Tower and Projector from Inkling 2 Small. The Tower was initialized from Meta's PE-Lang L and trained further with the language model.

Contents

File Tensors What it holds
model.safetensors 420 Tower, the parent's model.visual.backbone.* with the prefix model.visual. removed
projector.safetensors 2 Projector, the parent's model.visual.proj_in and proj_out
inkling2_vision.py Loads the Tower onto timm's vit_pe_lang_large_patch14_448 at patch size 20, plus the 2x2 merge, Projector and pixel normalization
config.json The parent's vision_config, the language model width and the native resolution
export_manifest.json Source pin, tensor counts and the vision reference
LICENSE Apache-2.0

Architecture

Component Details
Tower 23 blocks, 1024 hidden, 16 heads, patch size 20, 2D RoPE, exact GELU, LayerScale
Token compression 2x2 spatial grouping of 1024-wide patches to 4096, no learned parameters
Projector Linear(4096, 4096), exact GELU, Linear(4096, 4608), no biases

The native resolution is 640 px, a 32 x 32 patch grid. There the Tower gives 1024 patch tokens of width 1024, the merge 256 tokens of width 4096, and the Projector 256 tokens of width 4608, the language model's hidden size.

Usage

See examples/inference.py for image feature extraction.

Validation

The parity check compares every released tensor with the pinned parent using torch.equal. It then compares the Tower depths, the merge and the Projector with the eager per-image reference in vLLM's Inkling 2 vision test, run on the parent's tensors.

Reproduction

The export script reads the model.visual.* tensors from the pinned parent and writes the original BF16 tensors.

Credits

Thinking Machines released the Inkling 2 Small weights. Meta released PE-Lang L and the Perception Encoder code. The vision forward follows the vLLM implementation.

License

Apache-2.0, as declared on the source model card.

Downloads last month
14
Safetensors
Model size
0.3B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for immanuelpeter/Inkling-2-Small-Vision

Finetuned
(9)
this model

Collection including immanuelpeter/Inkling-2-Small-Vision