Update README.md

README.md

@@ -1,12 +1,13 @@
 ---
 title: README
-emoji: 👀
-colorFrom: red
-colorTo: pink
+emoji: 👁️
+colorFrom: purple
+colorTo: indigo
 sdk: static
 pinned: false
 license: apache-2.0
 ---
+
 <div align="center">
 
 <br>
@@ -15,101 +16,191 @@ license: apache-2.0
 
 <br><br>
 
+<img src="https://img.shields.io/badge/OpenLLaVA-v3.0.0-7C4DFF?style=for-the-badge&labelColor=0A0A0A" alt="v3.0.0">
+&nbsp;
+<img src="https://img.shields.io/badge/License-Apache--2.0-7C4DFF?style=for-the-badge&labelColor=0A0A0A" alt="License">
+&nbsp;
+<img src="https://img.shields.io/badge/Python-3.10+-3776AB?style=for-the-badge&labelColor=0A0A0A&logo=python&logoColor=3776AB" alt="Python">
+&nbsp;
+<img src="https://img.shields.io/badge/PyTorch-2.3+-EE4C2C?style=for-the-badge&labelColor=0A0A0A&logo=pytorch&logoColor=EE4C2C" alt="PyTorch">
+
+<br><br>
+
+<img src="https://img.shields.io/badge/CUDA-8.0%2B-76B900?style=for-the-badge&labelColor=0A0A0A&logo=nvidia&logoColor=76B900" alt="CUDA">
+&nbsp;
+<img src="https://img.shields.io/badge/ROCm-AMD-ED2B23?style=for-the-badge&labelColor=0A0A0A" alt="ROCm">
+&nbsp;
+<img src="https://img.shields.io/badge/TPU-Google-4285F4?style=for-the-badge&labelColor=0A0A0A" alt="TPU">
+&nbsp;
+<img src="https://img.shields.io/badge/MLX-Apple-555555?style=for-the-badge&labelColor=0A0A0A&logo=apple&logoColor=white" alt="MLX">
+&nbsp;
+<img src="https://img.shields.io/badge/XPU-Intel-0071C5?style=for-the-badge&labelColor=0A0A0A&logo=intel&logoColor=0071C5" alt="XPU">
+
+<br><br>
+
 # Inject Vision Into Any Language Model.
 
 **Open-source framework for adding multimodal vision capabilities to any HuggingFace LLM.**<br>
-**Low-level. Fast. Free. Built by [OpceanAI](https://huggingface.co/OpceanAI).**
+**Architecture-agnostic. Multi-backend. Production-ready. Built by [OpceanAI](https://huggingface.co/OpceanAI).**
 
+<br>
 
----
+[![GitHub](https://img.shields.io/badge/GitHub-OpceanAI%2Fopenllava-0D1117?style=for-the-badge&logo=github)](https://github.com/OpceanAI/openllava)
+&nbsp;
+[![HuggingFace](https://img.shields.io/badge/Models-Hugging_Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/Openllava)
+&nbsp;
+[![Sponsor](https://img.shields.io/badge/Sponsor-GitHub_Sponsors-ea4aaa?style=for-the-badge&logo=githubsponsors&logoColor=white)](https://github.com/sponsors/aguitauwu)
 
 <br>
 
+---
+
 </div>
 
 ## What is OpenLLaVA?
 
-**OpenLLaVA** is an open-source framework that injects vision capabilities into any language model — no architecture restrictions, no hardcoded backends, no compromises. Built on the LLaVA-style projection architecture and extended with custom CUDA kernels, a C++ core, and a clean Python API.
+**OpenLLaVA** is a comprehensive open-source framework for injecting vision capabilities into any language model. It provides a complete pipeline — from model construction through training, inference, serving, export, and evaluation — all accessible through a unified Python API and CLI.
 
-The framework is developed and maintained by **OpceanAI** as infrastructure for their vision model pipeline. Every model OpceanAI releases through OpenLLaVA feeds improvements back into the framework.
+The framework supports any LLM architecture (Llama, Mistral, Qwen, Gemma, Phi, DeepSeek, and more) and any HuggingFace-compatible vision encoder. It automatically detects model dimensions, constructs the appropriate projector, patches the tokenizer with visual tokens, and configures the full training and inference pipelines.
 
 The central design goal: **when a new language model drops, you should have a vision version in 48 hours.**
 
-<br>
-
----
+> OpenLLaVA is backend-agnostic. The same code runs on CUDA, ROCm, Apple MLX, Intel XPU, Google TPU, and CPU — with automatic hardware detection and optimal configuration selection.
 
 <br>
 
-<div align="center">
+---
 
 ## Quickstart
 
-</div>
-
-<br>
-
 ```bash
-pip install openllava
+pip install openllava        # Core
+pip install openllava[cli]   # With CLI tools
+pip install openllava[serve] # With serving
+pip install openllava[all]   # Full installation
 ```
 
-```python
-from openllava import patch_model
-from transformers import AutoModelForCausalLM, AutoTokenizer
+### Inject Vision Into Any LLM
 
-# Any HuggingFace model. Any vision encoder.
-model = AutoModelForCausalLM.from_pretrained("your-org/your-llm")
-tokenizer = AutoTokenizer.from_pretrained("your-org/your-llm")
+```python
+from openllava import OpenLLaVA, Backend
 
-model = patch_model(
-    model,
+model = OpenLLaVA(
+    llm="meta-llama/Llama-3-8B",
     vision_encoder="google/siglip2-so400m-patch14-384",
-    projector_layers=3,
+    backend=Backend.AUTO,
 )
 ```
 
 That's it. OpenLLaVA auto-detects hidden dimensions, builds the projector, and patches the tokenizer. No boilerplate. No config files.
 
-<br>
+### Train with LoRA
 
----
+```python
+model.lora(r=64, alpha=128, dropout=0.05)
 
-<br>
+model.train(
+    phase1=dict(dataset="liuhaotian/LLaVA-Pretrain", samples=100_000),
+    phase2=dict(dataset="liuhaotian/LLaVA-Instruct-150K", learning_rate=2e-4),
+    resume=True,
+)
 
-<div align="center">
+model.push("my-org/my-vision-model")
+```
 
-## Architecture
+### FastVisionModel API
 
-</div>
+```python
+from openllava.api import FastVisionModel
 
-<br>
+model, tokenizer = FastVisionModel.from_pretrained(
+    "Openllava/Yaki",
+    max_seq_length=2048,
+    load_in_4bit=True,
+)
 
-<table>
-<tr>
-<td width="50%" valign="top">
+model = FastVisionModel.get_peft_model(model, r=16, alpha=32)
+```
 
-**Vision Encoder**
+### Serve as OpenAI-Compatible API
 
-Any encoder from HuggingFace — SigLIP 2, CLIP, EVA-CLIP, InternViT. OpenLLaVA auto-reads the output dimension and handles tokenization regardless of encoder architecture.
+```bash
+openllava serve Openllava/Yaki --port 8000
+```
+
+```python
+from openai import OpenAI
+
+client = OpenAI(api_key="openllava", base_url="http://localhost:8000/v1")
+
+response = client.chat.completions.create(
+    model="yaki",
+    messages=[{
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "What is in this image?"},
+            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
+        ],
+    }],
+)
+```
 
 <br>
 
-**Projector Engine**
+---
 
-3-layer MLP with GELU activation, implemented as a fused CUDA kernel. Faster than PyTorch naive by design. Hidden dimension auto-computed from encoder output → LLM input.
+## Key Features
 
-</td>
+<table>
+<tr>
 <td width="50%" valign="top">
 
-**Model Patcher**
-
-Patches any HuggingFace causal LM to accept vision tokens. Adds `<image>` special token, extends the embedding layer, and wires the projector output into the LLM input stream. Supports LoRA-patched models.
+**Model Construction**
+- Vision injection into any HuggingFace LLM in 3 lines
+- AnyRes dynamic high-resolution with patch grouping
+- YakiProjector: configurable MLP alignment
+- Auto-detects hidden dims, attention heads, vocab size
+- Supports LoRA-patched models
+
+**Training Pipeline**
+- 3-phase training: alignment → instruction → RL
+- LoRA, LoRA+, DoRA, QLoRA, Split LoRA, LoRAGA, LoRAFA
+- BitNet ternary training (b1.58)
+- MoE + LoRA fusion
+- FP8 training on H100
+- Padding-free + sequence packing
+- Curriculum learning
+
+**RL Alignment**
+- DPO, GRPO, ORPO, PPO
+- Composable reward functions
+- Visual reasoning reward support
 
-<br>
-
-**Training Engine**
+</td>
+<td width="50%" valign="top">
 
-Two-phase training built in. Phase 1: projector warmup with frozen LLM. Phase 2: joint fine-tuning with LoRA. Gradient checkpointing, Flash Attention 2, and bfloat16 enabled by default.
+**Inference & Serving**
+- Continuous batching
+- PagedAttention (4x memory efficiency)
+- Speculative decoding (Eagle, Medusa, NGram)
+- KV cache: quantization, eviction, compression
+- OpenAI-compatible FastAPI server
+- Streaming support
+
+**40+ Optimizations**
+- torch.compile full-graph
+- GPTQ / AWQ / FP4 / NVFP4
+- GaLore gradient projection
+- torchao integration
+- EMA training stability
+- Selective activation checkpointing
+
+**Distributed Training**
+- FSDP2, DeepSpeed ZeRO (0-3)
+- Tensor, Pipeline, Expert parallelism
+- Ring Attention (long context)
+- Heterogeneous GPU+CPU+TPU training
+- Auto-parallelism detection
 
 </td>
 </tr>
@@ -119,94 +210,135 @@ Two-phase training built in. Phase 1: projector warmup with frozen LLM. Phase 2:
 
 ---
 
-<br>
+## Multi-Backend Support
 
-<div align="center">
+| Backend | Hardware | Status |
+|:--------|:---------|:-------|
+| CUDA | NVIDIA GPUs (Ampere, Ada, Hopper, Blackwell) | ✅ Production |
+| ROCm | AMD GPUs (MI250, MI300X, RX 7000) | ✅ Production |
+| CPU FP32 | Any x86/x64 CPU (AVX-512, AVX2, NEON) | ✅ Production |
+| TPU (XLA/SPMD) | Google TPU v3-v5 | 🔶 Beta |
+| MLX | Apple Silicon M1-M4 | 🔶 Beta |
+| XPU | Intel Arc, Data Center GPU | 🔶 Beta |
+| Heterogeneous | GPU + CPU + TPU mixed | 🔶 Beta |
 
-## Stack
+<br>
 
-</div>
+---
 
-<br>
+## Stack
 
 | Layer | Technology | Purpose |
 |:------|:----------:|:--------|
-| CUDA Kernels | C/CUDA | Fused projector ops, vision token attention |
-| Core | C++ | Memory management, tensor routing |
+| CUDA Kernels | C/CUDA | Fused projector ops, cross-attention, VQ lookup |
+| Core | C++ | Memory management, tensor routing, async streams |
 | Bindings | pybind11 | C++ → Python bridge |
-| API | Python | Public interface |
-| Export | HuggingFace | Standard model format + GGUF |
+| Triton | OpenAI Triton | Fused attention, RoPE, SwiGLU, RMSNorm |
+| API | Python | Public interface, FastVisionModel, Trainer |
+| Backends | CUDA/ROCm/MLX/TPU/XPU | Hardware abstraction |
+| Export | GGUF/ONNX/SafeTensors/vLLM/MLX | Deployment formats |
 
 <br>
 
 ---
 
-<br>
+## Architecture
 
-<div align="center">
+```
+┌─────────────────────────────────────────────────────────────┐
+│                      OpenLLaVA Framework                    │
+├───────────────────────────────────���─────────────────────────┤
+│                                                             │
+│  Input: Image + Text                                        │
+│         │                                                   │
+│  ┌──────▼──────────────────────────────────────────────┐   │
+│  │  Vision Encoder (SigLIP2, CLIP, DINOv2, any HF)     │   │
+│  └──────────────────────┬───────────────────────────────┘   │
+│                         │ patch features                    │
+│  ┌──────────────────────▼───────────────────────────────┐   │
+│  │  YakiProjector — Patch Grouping 3×3 + MLP 2-layer    │   │
+│  │  [vision_dim × 9] → [llm_dim]                        │   │
+│  └──────────────────────┬───────────────────────────────┘   │
+│                         │ vision embeddings                 │
+│  ┌──────────────────────▼───────────────────────────────┐   │
+│  │  Language Model (any AutoModelForCausalLM)            │   │
+│  │  QLoRA 4-bit NF4 · LoRA r=64 · Flash Attention        │   │
+│  └──────────────────────┬───────────────────────────────┘   │
+│                         │                                   │
+│  Output: Text + <think> reasoning blocks                    │
+└─────────────────────────────────────────────────────────────┘
+```
 
-## Training Pipeline
+<br>
 
-</div>
+---
 
-<br>
+## Yadis Architecture
+
+Yadis is OpenLLaVA's flagship multimodal architecture — the long-term evolution of the framework combining discrete visual tokens, MLP projection, and cross-attention per LLM layer.
 
 ```python
-from openllava import OpenLLaVATrainer
+# Yadis Routing — multiple vision experts with MoE router
+from openllava import OpenLLaVA, experts
+
+model = OpenLLaVA(
+    llm="OpceanAI/OwO-32B",
+    architecture="yadis_routing",
+    experts=[
+        experts.Visual("google/siglip2-so400m-patch14-384"),
+        experts.OCR("deepseek-ai/DeepSeek-OCR-2"),
+    ],
+)
 
-trainer = OpenLLaVATrainer(
-    model=model,
+# Yadis Full — discrete tokens + cross-attention per layer
+model = OpenLLaVA(
+    llm="OpceanAI/OwO-32B",
+    architecture="yadis_full",
     vision_encoder="google/siglip2-so400m-patch14-384",
-    pretrain_dataset="liuhaotian/LLaVA-Pretrain",   # Phase 1
-    instruct_dataset="liuhaotian/LLaVA-Instruct-150K",  # Phase 2
-    lora_r=64,
-    lora_alpha=128,
-    lora_target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
 )
-
-trainer.train()  # Handles both phases automatically
 ```
 
-OpenLLaVA manages phase transitions, learning rate schedules, and checkpoint saving. You run one command.
+| Mode | Description |
+|------|-------------|
+| `llava` | LLaVA-style MLP projection (default) |
+| `yadis_routing` | Multiple expert encoders + MoE router |
+| `yadis_full` | Discrete tokens + cross-attention per layer |
 
 <br>
 
 ---
 
-<br>
-
-<div align="center">
-
 ## OpceanAI Vision Models
 
-</div>
-
-<br>
-
-OpceanAI uses OpenLLaVA to publish vision versions of new language models as they release. These are the models built with the framework:
-
-<br>
+OpceanAI uses OpenLLaVA to publish vision versions of new language models within 48 hours of release.
 
 <table>
 <tr>
-<td width="50%" valign="top">
+<td width="33%" valign="top">
 
-**Yaki YuuKi+ Vision** *(in development)*
+**Yaki v1**
 
-Vision-language model built on Yuuki RxG 8B (DeepSeek-R1-Qwen2.5-8B fine-tune). Complex visual reasoning, bilingual (ES/EN), preserves the Yuuki `<think>` chain-of-thought behavior for multimodal tasks.
+Vision-language model on Yuuki RxG 8B. Complex visual reasoning, bilingual ES/EN, preserves `<think>` chain-of-thought for multimodal tasks.
 
-Vision encoder: SigLIP 2 SO400M · LoRA r=64
+Base: DeepSeek-R1-Qwen3-8B finetune<br>
+Encoder: SigLIP 2 SO400M<br>
+LoRA: r=64, alpha=128
 
-[![Status](https://img.shields.io/badge/Status-In_Development-orange?style=flat-square)](https://huggingface.co/OpceanAI)
+[![Status](https://img.shields.io/badge/Status-Training-orange?style=flat-square)](https://huggingface.co/Openllava/Yaki)
 
 </td>
-<td width="50%" valign="top">
+<td width="33%" valign="top">
+
+**Yaki v2** *(planned)*
+
+Built on Yuuki ExG 14B with cross-attention architecture (OpenLLaVA v4).
 
-**Yuuki NxG VL**
+</td>
+<td width="33%" valign="top">
 
-7B vision-language model fine-tuned from Qwen2.5-VL-7B-Instruct. Extends the NxG model family to multimodal tasks. The first OpceanAI vision model and the validation case for the OpenLLaVA pipeline.
+**Yaki v3** *(planned)*
 
-[![Model](https://img.shields.io/badge/Yuuki_NxG_VL-HuggingFace-ffd21e?style=flat-square&logo=huggingface&logoColor=black)](https://huggingface.co/OpceanAI/Yuuki-NxG-vl)
+Built on OwO 32B with full Yadis routing architecture. OCR + visual experts.
 
 </td>
 </tr>
@@ -216,45 +348,26 @@ Vision encoder: SigLIP 2 SO400M · LoRA r=64
 
 ---
 
-<br>
-
-<div align="center">
-
 ## Philosophy
 
-</div>
-
-<br>
-
 <table>
 <tr>
 <td width="50%" valign="top">
 
-**Model-Agnostic by Design**
+**Architecture Agnostic by Design**
 
-Every major framework for multimodal training — LLaVA, LLaVA-Next, InstructBLIP — is hardcoded to specific model families. OpenLLaVA is not. The projector adapts to any hidden dimension. The patcher works on any causal LM. The training engine handles any tokenizer.
-
-</td>
-<td width="50%" valign="top">
+Every existing multimodal framework is hardcoded to specific model families. OpenLLaVA is not. The projector adapts to any hidden dimension. The patcher works on any causal LM. The training engine handles any tokenizer.
 
 **Speed Over Ceremony**
 
-When a new language model drops, the window to publish a vision version is 48–72 hours before the ecosystem moves on. OpenLLaVA is designed for that constraint — minimal configuration, automated phase management, one-command training.
+When a new model drops, the window to publish a vision version is 48–72 hours. OpenLLaVA is designed for that constraint — minimal configuration, automated phase management, one-command training.
 
 </td>
-</tr>
-</table>
-
-<table>
-<tr>
 <td width="50%" valign="top">
 
 **Low Level Where It Matters**
 
-The projector is the critical path. Everything else can be Python. The CUDA kernel for the fused MLP op and the C++ memory manager exist because training throughput on a single A100 is the binding constraint for a zero-budget lab.
-
-</td>
-<td width="50%" valign="top">
+The projector is the critical path. The CUDA kernel for the fused MLP and the C++ memory manager exist because training throughput on a single GPU is the binding constraint for a zero-budget lab.
 
 **Fully Open**
 
@@ -268,94 +381,31 @@ Apache 2.0. No gating. No commercial restrictions. The framework exists so that
 
 ---
 
-<br>
-
-<div align="center">
-
 ## Roadmap
 
-</div>
-
-<br>
-
-<table>
-<tr>
-<td width="50%" valign="top">
-
-**Framework**
-
-| Feature | Status |
-|:--------|:------:|
-| Python API + model patcher | In development |
-| MLP projector (PyTorch) | In development |
-| Two-phase training engine | In development |
-| Fused CUDA projector kernel | Planned |
-| C++ memory core | Planned |
-| GGUF vision export | Planned |
-| Multi-encoder support (BRAVE-style) | Planned |
-
-</td>
-<td width="50%" valign="top">
-
-**Vision Models**
-
-| Model | Status |
-|:------|:------:|
-| Yuuki NxG VL | Released |
-| Yaki YuuKi+ Vision (8B) | In development |
-| Community model pipeline | Planned |
-
-</td>
-</tr>
-</table>
+| Version | Features | Status |
+|---------|----------|--------|
+| **v1-v3** | LLaVA-style, QLoRA, AnyRes, 3-phase pipeline, multi-backend | ✅ Done |
+| **v4-v5** | CUDA kernels, GGUF vision export, CPU offloading, cross-attention | 🔄 Active |
+| **v6-v7** | Discrete visual tokens (VQ-VAE), multi-expert routing | 📋 Planned |
+| **v8-v9** | Video support, hybrid architectures | 📋 Planned |
+| **v10** | Yadis complete, omnimodal prep | 📋 Planned |
 
 <br>
 
 ---
 
-<br>
-
-<div align="center">
-
-## Contributing
-
-</div>
-
-<br>
-
-OpenLLaVA is built to be extended. If you patch a model family that isn't supported yet, the contribution belongs in the framework. If you find a faster kernel implementation, open a PR.
-
-The project is maintained by OpceanAI but owned by the community.
-
-```bash
-git clone https://github.com/OpceanAI/openllava
-cd openllava
-pip install -e ".[dev]"
-```
-
-<br>
-
----
-
-<br>
-
 <div align="center">
 
 ## Built by OpceanAI
 
-</div>
-
-<br>
-
-<div align="center">
-
-OpenLLaVA is the vision infrastructure layer of [OpceanAI](https://huggingface.co/OpceanAI) — an independent AI research organization operating with no institutional funding, no cloud compute budget, and no team. Every model in the OpceanAI vision pipeline is trained on Google Colab Pro and validated on consumer hardware.
+OpenLLaVA is the vision infrastructure layer of [OpceanAI](https://huggingface.co/OpceanAI) — an independent AI research organization built from zero budget, consumer hardware, and measurable results.
 
 <br>
 
 [![OpceanAI](https://img.shields.io/badge/OpceanAI-Research-0D1117?style=for-the-badge)](https://huggingface.co/OpceanAI)
 &nbsp;
-[![HuggingFace](https://img.shields.io/badge/Models-Hugging_Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/OpceanAI)
+[![GitHub](https://img.shields.io/badge/GitHub-OpceanAI-0D1117?style=for-the-badge&logo=github)](https://github.com/OpceanAI/openllava)
 &nbsp;
 [![Sponsor](https://img.shields.io/badge/Sponsor-GitHub_Sponsors-ea4aaa?style=for-the-badge&logo=githubsponsors&logoColor=white)](https://github.com/sponsors/aguitauwu)
 
@@ -363,16 +413,10 @@ OpenLLaVA is the vision infrastructure layer of [OpceanAI](https://huggingface.c
 
 ---
 
-<br>
-
 **Open framework. Open models. Zero budget. Measurable results.**
 
-<br>
-
-[![OpenLLaVA](https://img.shields.io/badge/OpenLLaVA-2026-0D1117?style=for-the-badge)](https://github.com/OpceanAI/openllava)
+[![OpenLLaVA](https://img.shields.io/badge/OpenLLaVA-v3.0.0-0D1117?style=for-the-badge)](https://github.com/OpceanAI/openllava)
 
-<br>
+*Inject vision into any language model.*
 
-*The fastest path from any language model to a vision-language model.*
-
-</div>
+</div>