--- language: en license: other library_name: transformers tags: - vision - vlm - vision-language-model - image-text - instruction-tuning - dpo - lora - custom-architecture pipeline_tag: image-text-to-text ---
# Keural VLM — Vision-Language Model (PoC · V0.1) **##Keural VLM is a proof-of-concept vision-language model developed to explore lightweight multimodal learning using a custom vision encoder. Unlike many existing VLMs, the model does not rely on a pretrained CLIP backbone. Instead, the visual encoder is trained from scratch and connected to Mistral-7B-Instruct through a lightweight projection module. .**

MKD Status

Params LLM PyTorch Transformers PEFT Hardware


Keural VLM architecture animation
*Image → CNN Stem → Adaptive Token Budget → Spatial Transformer → LevelAware Projector → Mistral-7B → Answer*
--- ## Highlights | | | |---|---| | **From scratch** | 24.7M vision encoder — **12.4× smaller** than LLaVA's CLIP encoder (307M) | | **Adaptive Token Budget (ATB)** | Token count is a runtime knob — dense regions get more tokens, blank regions fewer | | **Hierarchical Concept Tokenization (HCT)** | Every token carries a semantic level: global / region / detail | | **3-phase pipeline** | Vision pretraining → projector alignment → SFT + DPO | | **DPO improved every benchmark** | e.g. VQAv2 **+30.7pp**, ScienceQA **+14.0pp** | --- ## What This Is A complete Vision-Language Model (VLM) proof-of-concept built entirely from scratch — no CLIP, no pretrained vision backbone. | Component | Details | |---|---| | **Vision Encoder** | 24.7M params, trained from scratch on CC3M + CC12M (~15M pairs) | | **Projector** | `LevelAwareProjector` (384 → 2048 → 4096) | | **LLM** | `Mistral-7B-Instruct-v0.3` (4-bit NF4 QLoRA) | | **SFT** | LLaVA-Instruct-150K, 30,000 steps | | **DPO** | RLHF-V dataset, 5,733 pairs, 3,000 steps | --- ## Architecture ``` Image (256×256) ↓ CNN Stem → ATB Tokenizer → Spatial Transformer (12 layers, embed_dim=384) ↓ KeuralEncoderOutput {tokens, level_ids, spatial_metadata, saliency_scores, pooled} ↓ LevelAwareProjector (384 → 2048 → 4096) ↓ Visual Tokens (N_vis × 4096) ↓ Mistral-7B-Instruct-v0.3 + SFT LoRA + DPO LoRA ↓ Text Response ```
Keural VLM detailed architecture
### Key Innovations **Adaptive Token Budget (ATB) Tokenization** — token count is a runtime parameter; dense regions get more tokens, blank regions fewer. ```python out = encoder(image, token_budget=64) # fast / cheap out = encoder(image, token_budget=256) # default out = encoder(image, token_budget=1024) # full fidelity ``` **Hierarchical Concept Tokenization (HCT)** — every token carries a semantic level tag. ```python out = encoder(image) print(out.level_ids) # {0=global, 1=region, 2=detail} ``` --- ## Training Pipeline
PhaseWhat trainsDataStepsResult
1 · Vision Encoder Encoder from scratchCC3M + CC12M (~15.3M)~75,00024.7M params · 1× RTX 5090
2A · Projector Align Projector + LLM LoRA (r=64), encoder frozenLLaVA-Instruct-150K10,000—
2B · SFT LLM LoRA (r=64, α=128)LLaVA-Instruct-150K30,000final loss 1.022
2B · DPO DPO LoRA (r=16, α=32)RLHF-V (5,733 pairs)3,000loss 0.235 · reward acc 95% · margin 2.11
--- ## Benchmark Results Evaluated on 1,000 samples each (where applicable). The vision encoder is **12.4× smaller** than LLaVA's CLIP encoder (307M). | Benchmark | Keural SFT-30K | Keural SFT+DPO | LLaVA 1.5 (307M enc) | LLaVA 1.6 (307M enc) | |---|:---:|:---:|:---:|:---:| | VQAv2 Accuracy | 12.9% | **43.6%** | 78.5% | 81.8% | | POPE F1 | 66.9% | **67.0%** | 85.9% | 86.5% | | MME Total Score | 704.3 | **838.8** | 1510.7 | 1519.3 | | TextVQA Accuracy | 0.8% | **6.6%** | 58.2% | 64.9% | | ScienceQA Accuracy | 39.7% | **53.7%** | 66.8% | 70.6% | > **POPE F1 (67.0%)** is the standout — within 19pp of LLaVA 1.6 using a **12× smaller** encoder. > **DPO improved every benchmark**, most dramatically VQAv2 (+30.7pp) and ScienceQA (+14.0pp). > **TextVQA** is low by design — no OCR training. EasyOCR integration in the GUI bridges this gap.
All benchmarks
| | | |:---:|:---:| | | | | | | --- ## Qualitative & Saliency The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement.
Saliency — cat Qualitative examples
--- ## DPO Training Curves | Metric | Step 0 | Step 3000 | |---|:---:|:---:| | Loss | 0.694 | **0.235** | | Reward Accuracy | ~50% | **95%** | | Reward Margin | 0.0 | **2.11** |
DPO training curves
--- ## Repository Structure ``` keural-vlm-poc/ ├── vision_encoder/ # Keural encoder weights (config + safetensors) ├── sft_adapter/ # SFT LoRA (30K steps) ├── dpo_adapter/ # DPO LoRA (3K steps, RLHF-V) — stacks on SFT ├── assets/ # figures, diagrams, animation └── tokenizer.json … # Mistral tokenizer + chat template ``` For inference, load: **Vision Encoder → Projector → Mistral-7B + SFT LoRA + DPO LoRA**. --- ## Usage ```python import torch from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from peft import PeftModel from alignment.projectors import LevelAwareProjector device = "cuda" # 1. Vision encoder (frozen) encoder = AutoModel.from_pretrained( "mkd-hika/keural-vision-encoder-poc", trust_remote_code=True, torch_dtype=torch.bfloat16, ).to(device).eval() # 2. Projector projector = LevelAwareProjector(encoder_dim=384, hidden_dim=2048, llm_dim=4096) projector.load_state_dict(torch.load("projector.pt", map_location=device)) projector = projector.to(device, dtype=torch.bfloat16).eval() # 3. LLM + SFT LoRA + DPO LoRA bnb_cfg = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4") base_llm = AutoModelForCausalLM.from_pretrained( "mistralai/Mistral-7B-Instruct-v0.3", quantization_config=bnb_cfg, device_map="auto", torch_dtype=torch.bfloat16, ) llm = PeftModel.from_pretrained(base_llm, "sft_adapter") # SFT LoRA llm = PeftModel.from_pretrained(llm, "dpo_adapter") # DPO LoRA llm.eval() ``` --- ## Roadmap | Phase | Encoder Params | Hardware | Status | |---|:---:|:---:|:---:| | **PoC** (this model) | 24.7M | 1× RTX 5090 | Complete (SFT + DPO) | | Mid-level | ~183.5 | 8× H100 80 GB | Planned | | Commercial | ~1.1B | 64× H100 80 GB | Future | --- ## Citation ```bibtex @misc{keural_vlm_2026, title = {Keural VLM: Vision-Language Model with Content-Adaptive Encoding via Saliency-Guided Token Budgets}, author = {Hika Barki and MKD Co., Ltd.}, year = {2026}, }, } ``` --- ## 📄 License See [LICENSE](LICENSE) for details. *Training data: CC3M, CC12M, LLaVA-Instruct-150K, RLHF-V — respective data licenses apply.*
**MKD Co., Ltd. — 2026**