license: apache-2.0
Model Summary
25EMBAI-VLM-FM is a Vision-Language Foundation Model built by combining:
Vision Encoder: ViT-H/14 (OpenCLIP)
Language Model: Qwen-based LLM
Bridging Modules: Resampler + Projector (image → LLM embedding space)
It takes an image, encodes it into patch tokens, compresses them into a fixed-length set of visual tokens, projects them into the language model’s hidden space, and then performs multimodal reasoning conditioned on a text prompt.
Architecture Flow
Image → ViT-H/14 → Resampler → Projector → Qwen LLM → Text Output LLM Input Format [Batch, K_image_tokens + T_text_tokens, D_hidden]
Training Summary
Pre-training (Stage 1 & 2)
Hardware: 8 × H100 80GB Stage 1 (3.6h): Freeze ViT + LLM → Train Resampler + Projector Stage 2 (5.4h): Unfreeze all → Train end-to-end Data: ~2M image–caption pairs (BLIP3 style)
Instruction Fine-tuning
~2M images + ~200M text tokens ~20 multimodal tasks: VQA, OCR, captioning, commands max_length: 1024 effective batch size: ~64
Usage
Install
pip install torch transformers pillow
Inference Example
from transformers import AutoModel, AutoTokenizer, AutoImageProcessor
import torch
from PIL import Image
model_path = '/home/raid/models/25EMBAI_save_test'
vision_model = 'ViT-H-14-378-quickgelu'
vision_pretrained = 'dfn5b'
dtype = torch.bfloat16
image_path = '/home/jason/git/UNIVA/25EMBAI_VLM_FM/qwen/train/sample.png'
model = AutoModel.from_pretrained(
model_path,
trust_remote_code=True
).to(device = 'cuda', dtype=dtype)
tokenizer = AutoTokenizer.from_pretrained(model_path)
image_processor = AutoImageProcessor.from_pretrained(
model_path,
trust_remote_code=True,
)
model.eval()
img = Image.open(image_path).convert("RGB")
pixel = image_processor(img, return_tensors="pt")["pixel_values"].to(
dtype=dtype,
device='cuda',
)
prompt = 'please describe this image.'
output = model.generate_text(
images=pixel,
prompt=prompt,
max_new_tokens=512,
do_sample=True,
top_p=0.9,
temperature=0.7,
)
print(output)
Limitations & Biases
This model is an early-stage prototype. It will be updated and reorganized in future releases. Because it was trained on web-scale multimodal data: It may reflect social biases and stereotypes It may hallucinate, invent facts, or produce unverifiable content It may perform suboptimally on: Non-English languages Specialized and domain-specific tasks Safety-critical contexts This model is not recommended for medical, legal, or safety-critical use without additional validation, guardrails, or fine-tuning. Users should apply external filtering, grounding, and safety alignment before deployment.