Luna-0.1b-VL (Vision-Language)

This is a 124 Million parameter Multimodal Vision-Language Model built from scratch. It extends the base text-only model by introducing a vision encoder and a learned projection layer, allowing it to perceive, understand, and reason about images.

🧠 Architecture Details

The model fuses text and vision using the following components:

1. Vision Encoder: google/siglip-base-patch16-224 (Extracts rich image features).
2. Vision Projector: A custom 3-layer MLP that projects the 768-dimensional visual features into the GPT-2 text embedding space.
3. Backend LLM: The custom 124M parameter GPT-2 Small architecture, fine-tuned to accept visual tokens alongside text.

📉 Multimodal Training

The model was trained on the LLaVA-Instruct dataset (complex reasoning split) to learn how to answer questions about images.

🏆 Multimodal Benchmarks

The model was evaluated against strict computer vision benchmarks to measure its perception, logic, and hallucination resistance.

Benchmark	Score	What it means
MME Perception (Accuracy)	58.00%	Binary Yes/No evaluation of visual perception. Solidly beats the 50% random chance baseline.
MMBench (Logic)	50.00%	4-choice visual logic test. Impressively doubles the 25% random chance baseline, proving visual reasoning capabilities!

💻 How to Load and Run

Because this uses custom multimodal injection mechanics, you must use the provided PyTorch script rather than the standard transformers pipeline. Ensure you download model.py, vqa_inference.py, vqa_model.safetensors, and vqa_projector.safetensors.

import torch
from transformers import AutoTokenizer, AutoModel, AutoProcessor
from model import GPTModel, VisionProjector, generate_text, text_to_token_ids, token_ids_to_text
import torch.nn as nn
from safetensors.torch import load_file
import json
from PIL import Image
import requests

device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained("tokenizer")

# 1. Load Vision Components
vision_model = AutoModel.from_pretrained("google/siglip-base-patch16-224").to(device)
image_processor = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")
vision_model.eval()

vision_projector = VisionProjector().to(device)
vision_projector.load_state_dict(load_file("vqa_projector.safetensors"))
vision_projector.eval()

# 2. Load GPT Backend
with open("tokenizer/config.json") as f:
    cfg = json.load(f)
model = GPTModel(cfg).to(device)
model.load_state_dict(load_file("vqa_model.safetensors"))
model.eval()

# 3. Process Image
raw_image = Image.open("llava_002509.jpg").convert("RGB")
image_tensor = image_processor(images=raw_image, return_tensors="pt")['pixel_values'].to(device)

with torch.no_grad():
    vision_outputs = vision_model.vision_model(pixel_values=image_tensor)
    raw_image_features = vision_outputs.pooler_output.unsqueeze(1)
    image_embeds = vision_projector(raw_image_features)

# 4. Generate
prompt = (
    "Below is an instruction that describes a task. "
    "Write a response that appropriately completes the request.\n\n"
    "### Instruction:\nWhat is in this image?\n\n### Response:\n"
)
encoded = text_to_token_ids(prompt, tokenizer).to(device)
token_ids = generate_text(
    model=model,
    idx=encoded,
    max_new_tokens=50,
    context_size=model.pos_emb.weight.shape[0],
    temperature=0.5,
    top_k=30,
    repetition_penalty=1.15,
    image_embeds=image_embeds
)
print(token_ids_to_text(token_ids, tokenizer).split("### Response:\n")[-1])

📝 Sample Output

Here is an example of the model correctly identifying objects and their attributes:

Prompt:

What is in this image?

Output:

The image features a pizza with various toppings, including cheese and pepperoni, placed on top of a plate.