Instructions to use immanuelpeter/Inkling-2-Small-Vision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- timm
How to use immanuelpeter/Inkling-2-Small-Vision with timm:
import timm model = timm.create_model("hf-hub:immanuelpeter/Inkling-2-Small-Vision", pretrained=True) - PerceptionEncoder
How to use immanuelpeter/Inkling-2-Small-Vision with PerceptionEncoder:
# Use any PE model as a vision encoder import core.vision_encoder.pe as pe model = pe.VisionTransformer.from_config("immanuelpeter/Inkling-2-Small-Vision", pretrained=True) - Notebooks
- Google Colab
- Kaggle
|
Download README.md from immanuelpeter/Inkling-2-Small-Vision: direct link, hf CLI and curl.
- Browser
- Download file 2.67 kB
-
https://huggingface.co/immanuelpeter/Inkling-2-Small-Vision/resolve/main/README.md
- Command line
-
hf download hf://immanuelpeter/Inkling-2-Small-Vision/README.md
-
curl -L -o README.md https://huggingface.co/immanuelpeter/Inkling-2-Small-Vision/resolve/main/README.md
2.67 kB
| license: apache-2.0 | |
| pipeline_tag: image-feature-extraction | |
| library_name: timm | |
| tags: | |
| - vision | |
| - feature-extraction | |
| - perception-encoder | |
| base_model: | |
| - thinkingmachines/Inkling-2-Small | |
| # Inkling-2-Small-Vision | |
| This repository packages the vision Tower and Projector from [Inkling 2 Small](https://huggingface.co/thinkingmachines/Inkling-2-Small). The Tower was initialized from Meta's [PE-Lang L](https://huggingface.co/facebook/vit_pe_lang_large_patch14_448_timm) and trained further with the language model. | |
| ## Contents | |
| | File | Tensors | What it holds | | |
| |---|---|---| | |
| | `model.safetensors` | 420 | Tower, the parent's `model.visual.backbone.*` with the prefix `model.visual.` removed | | |
| | `projector.safetensors` | 2 | Projector, the parent's `model.visual.proj_in` and `proj_out` | | |
| | `inkling2_vision.py` | | Loads the Tower onto timm's `vit_pe_lang_large_patch14_448` at patch size 20, plus the 2x2 merge, Projector and pixel normalization | | |
| | `config.json` | | The parent's `vision_config`, the language model width and the native resolution | | |
| | `export_manifest.json` | | Source pin, tensor counts and the vision reference | | |
| | `LICENSE` | | Apache-2.0 | | |
| ## Architecture | |
| | Component | Details | | |
| |---|---| | |
| | Tower | 23 blocks, 1024 hidden, 16 heads, patch size 20, 2D RoPE, exact GELU, LayerScale | | |
| | Token compression | 2x2 spatial grouping of 1024-wide patches to 4096, no learned parameters | | |
| | Projector | `Linear(4096, 4096)`, exact GELU, `Linear(4096, 4608)`, no biases | | |
| The native resolution is 640 px, a 32 x 32 patch grid. There the Tower gives 1024 patch tokens of width 1024, the merge 256 tokens of width 4096, and the Projector 256 tokens of width 4608, the language model's hidden size. | |
| ## Usage | |
| See [`examples/inference.py`](examples/inference.py) for image feature extraction. | |
| ## Validation | |
| The [parity check](https://github.com/immanuel-peter/vision-tower-bench/blob/main/vtb/parity/pe.py) compares every released tensor with the pinned parent using `torch.equal`. It then compares the Tower depths, the merge and the Projector with the eager per-image reference in vLLM's Inkling 2 vision [test](https://github.com/vllm-project/vllm/pull/67325), run on the parent's tensors. | |
| ## Reproduction | |
| The [export script](https://github.com/immanuel-peter/vision-tower-bench/blob/main/vtb/adapters/pe.py) reads the `model.visual.*` tensors from the pinned parent and writes the original BF16 tensors. | |
| ## Credits | |
| Thinking Machines released the Inkling 2 Small weights. Meta released PE-Lang L and the Perception Encoder code. The vision forward follows the vLLM implementation. | |
| ## License | |
| Apache-2.0, as declared on the source model card. | |