|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# Model Summary |
|
|
## 25EMBAI-VLM-FM is a Vision-Language Foundation Model built by combining: |
|
|
### Vision Encoder: ViT-H/14 (OpenCLIP) |
|
|
### Language Model: Qwen-based LLM |
|
|
### Bridging Modules: Resampler + Projector (image → LLM embedding space) |
|
|
|
|
|
It takes an image, encodes it into patch tokens, compresses them into a fixed-length set of visual tokens, projects them into the language model’s hidden space, and then performs multimodal reasoning conditioned on a text prompt. |
|
|
|
|
|
## Architecture Flow |
|
|
Image → ViT-H/14 → Resampler → Projector → Qwen LLM → Text Output |
|
|
LLM Input Format
[Batch, K_image_tokens + T_text_tokens, D_hidden] |
|
|
|
|
|
## Training Summary |
|
|
### Pre-training (Stage 1 & 2) |
|
|
Hardware: 8 × H100 80GB |
|
|
Stage 1 (3.6h):
Freeze ViT + LLM → Train Resampler + Projector |
|
|
Stage 2 (5.4h):
Unfreeze all → Train end-to-end |
|
|
Data: ~2M image–caption pairs (BLIP3 style) |
|
|
### Instruction Fine-tuning |
|
|
~2M images + ~200M text tokens |
|
|
~20 multimodal tasks: VQA, OCR, captioning, commands |
|
|
max_length: 1024 |
|
|
effective batch size: ~64 |
|
|
|
|
|
|
|
|
# Usage |
|
|
|
|
|
## Install |
|
|
pip install torch transformers pillow |
|
|
|
|
|
## Inference Example |
|
|
``` |
|
|
from transformers import AutoModel, AutoTokenizer, AutoImageProcessor |
|
|
import torch |
|
|
from PIL import Image |
|
|
|
|
|
model_path = '/home/raid/models/25EMBAI_save_test' |
|
|
vision_model = 'ViT-H-14-378-quickgelu' |
|
|
vision_pretrained = 'dfn5b' |
|
|
dtype = torch.bfloat16 |
|
|
image_path = '/home/jason/git/UNIVA/25EMBAI_VLM_FM/qwen/train/sample.png' |
|
|
|
|
|
model = AutoModel.from_pretrained( |
|
|
model_path, |
|
|
trust_remote_code=True |
|
|
).to(device = 'cuda', dtype=dtype) |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_path) |
|
|
image_processor = AutoImageProcessor.from_pretrained( |
|
|
model_path, |
|
|
trust_remote_code=True, |
|
|
) |
|
|
model.eval() |
|
|
|
|
|
img = Image.open(image_path).convert("RGB") |
|
|
|
|
|
pixel = image_processor(img, return_tensors="pt")["pixel_values"].to( |
|
|
dtype=dtype, |
|
|
device='cuda', |
|
|
) |
|
|
prompt = 'please describe this image.' |
|
|
|
|
|
output = model.generate_text( |
|
|
images=pixel, |
|
|
prompt=prompt, |
|
|
max_new_tokens=512, |
|
|
do_sample=True, |
|
|
top_p=0.9, |
|
|
temperature=0.7, |
|
|
) |
|
|
|
|
|
print(output) |
|
|
``` |
|
|
|
|
|
# Limitations & Biases |
|
|
This model is an early-stage prototype.
It will be updated and reorganized in future releases. |
|
|
Because it was trained on web-scale multimodal data: |
|
|
It may reflect social biases and stereotypes |
|
|
It may hallucinate, invent facts, or produce unverifiable content |
|
|
It may perform suboptimally on: |
|
|
Non-English languages |
|
|
Specialized and domain-specific tasks |
|
|
Safety-critical contexts |
|
|
This model is not recommended for medical, legal, or safety-critical use without additional validation, guardrails, or fine-tuning. |
|
|
Users should apply external filtering, grounding, and safety alignment before deployment. |
|
|
|
