Models
Vision Backbone (ViT)
This is a very lightweight Vision Transformer in native pytorch. I took inspiration from the following sources:
- https://github.com/karpathy/nanoGPT (General Transformer Decoder)
- https://arxiv.org/abs/2010.11929 (ViT Paper)
- https://arxiv.org/abs/2303.15343 (SigLiP Paper)
- https://github.com/huggingface/transformers/blob/main/src/transformers/models/siglip/modeling_siglip.py (HF SigLiP Implementation)
Language Model (Llama / SmolLM)
This is a decoder only LM, following the Llama 2/3 architecture. Inspiration from the following sources:
- https://arxiv.org/pdf/2307.09288 (Original Llama Paper)
- https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py (HF Llama Implementation)
Modality Projection
This is a simple MLP (Linear Layer) for the Modality Projection between the Image Patch Encodings and the Language Embedding Space with a simple Pixel Shuffle (https://arxiv.org/pdf/2504.05299)
Vision-Language-Model
This brings all the individual parts together and handles the concatenation of images and text. Built as a simple version of SmolVLM (https://arxiv.org/pdf/2504.05299)