RainbowPiBubbles/Nathanael-Embedding-50M

Encoder-Free Vision-Language Model (VLM)

This repository hosts an Encoder-Free Vision-Language Model (VLM) wrapped around the base language model SupraLabs/Supra-1.5-50M-Base-exp and distilled using embeddings generated by SigLIP-2.

By using robust mathematical hook patches instead of standard heavy vision encoders, it extracts and maps visual tokens directly into target hidden representation vectors in the browser or terminal.

File Registry

• vlm_model.onnx: Optimized FP32 ONNX model compatible with CPU/WASM onnxruntime-web execution providers.
• vlm_model_fp16.onnx: Optimized FP16 ONNX model for WebGPU/WebGL rendering acceleration.
• model.safetensors: Standard PyTorch model state dictionary (SafeTensors format).
• modeling_vlm.py: Custom Python wrapper code for the EncoderFreeVLM module and VLMPreprocessor.
• config.json: Hardware parameter settings.

PyTorch Integration

To load this model natively in Python, clone this repository and use the custom classes in modeling_vlm.py:

import torch
from modeling_vlm import EncoderFreeVLM, VLMPreprocessor
from transformers import AutoTokenizer, AutoModelForCausalLM

# 1. Load base components
tokenizer = AutoTokenizer.from_pretrained("SupraLabs/Supra-1.5-50M-Base-exp", trust_remote_code=True)
cma_model = AutoModelForCausalLM.from_pretrained("SupraLabs/Supra-1.5-50M-Base-exp", trust_remote_code=True)

# 2. Instantiate wrapper
preprocessor = VLMPreprocessor(tokenizer)
vlm = EncoderFreeVLM(cma_model, embedding_dim=768, proj_mode="linear")

# 3. Load model state dictionary and automatically reconstruct shared weights
from safetensors.torch import load_model
load_model(vlm, "model.safetensors")
vlm.eval()

# Example forward execution
# inputs = preprocessor(image=your_image_object)
# embeddings = vlm(**inputs)

ONNX Runtime Configuration

When deploying inside JS/Web applications, query the model with the following inputs:

• input_ids [int64, [batch_size, sequence_length]]
• attention_mask [int64, [batch_size, sequence_length]]
• images [float32, [batch_size, 3, 224, 224]]
• is_image [float32, [batch_size, 1]]