mkd-hika/keural-vision-encoder-poc

👁️ Keural Vision Encoder _{(PoC · V0.1)}

A 24.7M-parameter vision encoder trained from scratch — no pretrained backbone, no CLIP weights. Structured tokens that plug into any LLM.

The encoder (CNN Stem → ATB → Spatial Transformer) feeds the LevelAware Projector → Mistral-7B in the full Keural VLM.

---

✨ Key Innovations

🎯 Adaptive Token Budget (ATB) Tokenization — token count is a runtime parameter. Tokens are allocated to information-dense regions: a blank wall gets fewer, a dense document gets more.

out = encoder(image, token_budget=64)    # cheap
out = encoder(image, token_budget=256)   # default
out = encoder(image, token_budget=1024)  # full fidelity

🪜 Hierarchical Concept Tokenization (HCT) — every token carries a semantic level tag: global (whole-image), region (object-scale), or detail (fine-grained).

out = encoder(image)
print(out.level_ids)  # tensor of {0=global, 1=region, 2=detail}

---

⚙️ Model Specifications

Property	Value
Parameters	24.7M (trained from scratch)
Architecture	CNN Stem + ATB Tokenizer + 12-layer Spatial Transformer
Embedding dim	384
Token budget	256 (default) · runtime-adjustable
Token split	5% global / 25% region / 70% detail
Precision	bfloat16
Training data	CC3M + CC12M (~6.9M image-text pairs)
Hardware	1× RTX 5090 (32 GB VRAM)

---

🎨 Saliency & Token Placement

The ATB tokenizer concentrates tokens on salient regions. Left → right: original · saliency heatmap · token placement (global / region / detail).

---

🧪 Training

Phase 1 — Vision Encoder Pretraining ✅ COMPLETE. Trained from scratch for 75,000 steps on CC3M + CC12M (6.9M image–text pairs), SigLIP-style contrastive objective, 1× RTX 5090. The frozen encoder is then integrated into the full VLM via LevelAwareProjector (384 → 2048 → 4096) → Mistral-7B-Instruct-v0.3 (4-bit NF4 QLoRA), fine-tuned with SFT (LLaVA-Instruct-150K, 30K steps).

---

📊 Benchmark Results — SFT-30K

Downstream VLM benchmarks using this encoder (frozen) + projector + Mistral-7B at the SFT-30K checkpoint (supervised fine-tuning, 30,000 steps; before DPO). Evaluated on 1,000 samples each where applicable. The Keural encoder is 12.4× smaller than LLaVA's CLIP encoder (307M).

Benchmark	Keural SFT-30K	LLaVA 1.5 (307M enc)	LLaVA 1.6 (307M enc)
VQAv2 Accuracy	12.9%	78.5%	81.8%
POPE F1	66.9%	85.9%	86.5%
MME Total Score	704.3	1510.7	1519.3
TextVQA Accuracy	0.8%	58.2%	64.9%
ScienceQA Accuracy (img)	39.7%	66.8%	70.6%

These are the SFT-30K numbers (measured from outputs/eval/sft_30k/). Applying DPO alignment (RLHF-V, 3K steps) improves every metric — e.g. VQAv2 → 43.6%, ScienceQA → 53.7%, MME → 838.8. Full SFT+DPO results, the comparison chart, and the complete VLM live on mkd-hika/keural-vlm-poc.

---

⚡ Adaptive Token Budget — latency vs budget

Encode latency vs token budget — trade visual detail for speed at runtime, no retraining.

---

📐 Output Contract

Every forward pass returns a KeuralEncoderOutput:

Field	Shape	Description
tokens	(B, N, 384)	Token vectors
attention_mask	(B, N)	1 = real token, 0 = padding
level_ids	(B, N)	0=global, 1=region, 2=detail
spatial_metadata	(B, N, 4)	Per-token (cx, cy, w, h) in [0,1]
saliency_scores	(B, N)	Allocator confidence per token
pooled	(B, 384)	Single-vector image summary
token_budget_used	(B,)	Actual tokens used per image

---

🚀 Usage

import torch
from architecture.encoder import KeuralVisionEncoder
from keural_config import KeuralConfig

cfg = KeuralConfig.from_yaml("configs/keural_tiny_poc.yaml")
model = KeuralVisionEncoder.from_pretrained("mkd-hika/keural-vision-encoder-poc")
model.eval()

from PIL import Image
from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((256, 256)),
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5]),
])

image = Image.open("image.jpg").convert("RGB")
x = transform(image).unsqueeze(0)  # (1, 3, 256, 256)

with torch.no_grad():
    out = model(x)

print(out.tokens.shape)        # (1, 256, 384)
print(out.level_ids[0, :8])    # [0, 0, 1, 1, 1, 2, 2, 2]
print(out.pooled.shape)        # (1, 384)

---

🗺️ Roadmap

Phase	Params	Hardware	Status
PoC (this model)	24.7M	1× RTX 5090	✅ Phase 1 complete
Mid-level	~230M	8× H100 80 GB	🔜 Planned
Commercial	~1.1B	64× H100 80 GB	🔮 Future

The mid-level model will use knowledge distillation from SigLIP-400M (natural images) and InternViT-300M (documents/OCR). Teachers are discarded after training — not part of the final model.

---

📝 Citation

@misc{keural_vision_encoder_2026,
  title  = {Keural Vision Encoder: Content-Adaptive Vision Encoding via Saliency-Guided Token Budgets},
  author = {Barki, Hika and MKD Co., Ltd.},
  year   = {2026},
  note   = {V0.1 — Phase 1 complete},
}

---

📄 License

Code: MIT. Model weights trained on CC3M + CC12M — data licenses apply.

MKD Co., Ltd. — 2026