Update README.md

README.md

@@ -5,6 +5,385 @@ colorFrom: red
 colorTo: pink
 sdk: static
 pinned: false
+license: apache-2.0
 ---
+<div align="center">
 
-Edit this `README.md` markdown file to author your organization card.
+<br>
+
+<img src="https://img.shields.io/badge/%F0%9F%91%81%EF%B8%8F-OPENLLAVA-0D1117?style=for-the-badge&labelColor=0D1117" alt="OpenLLaVA" height="60">
+
+<br><br>
+
+# Inject Vision Into Any Language Model.
+
+**Open-source framework for adding multimodal vision capabilities to any HuggingFace LLM.**<br>
+**Low-level. Fast. Free. Built by [OpceanAI](https://huggingface.co/OpceanAI).**
+
+<br>
+
+[![PyPI](https://img.shields.io/badge/PyPI-openllava-3775a9?style=for-the-badge&logo=pypi&logoColor=white)](https://pypi.org/project/openllava)
+&nbsp;
+[![HuggingFace](https://img.shields.io/badge/Models-Hugging_Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/OpenLLaVA)
+&nbsp;
+[![License](https://img.shields.io/badge/License-Apache_2.0-green?style=for-the-badge)](https://opensource.org/licenses/Apache-2.0)
+&nbsp;
+[![Discord](https://img.shields.io/badge/Discord-Community-5865F2?style=for-the-badge&logo=discord&logoColor=white)](https://discord.gg/openllava)
+
+<br>
+
+---
+
+<br>
+
+</div>
+
+## What is OpenLLaVA?
+
+**OpenLLaVA** is an open-source framework that injects vision capabilities into any language model — no architecture restrictions, no hardcoded backends, no compromises. Built on the LLaVA-style projection architecture and extended with custom CUDA kernels, a C++ core, and a clean Python API.
+
+The framework is developed and maintained by **OpceanAI** as infrastructure for their vision model pipeline. Every model OpceanAI releases through OpenLLaVA feeds improvements back into the framework.
+
+The central design goal: **when a new language model drops, you should have a vision version in 48 hours.**
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## Quickstart
+
+</div>
+
+<br>
+
+```bash
+pip install openllava
+```
+
+```python
+from openllava import patch_model
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Any HuggingFace model. Any vision encoder.
+model = AutoModelForCausalLM.from_pretrained("your-org/your-llm")
+tokenizer = AutoTokenizer.from_pretrained("your-org/your-llm")
+
+model = patch_model(
+    model,
+    vision_encoder="google/siglip2-so400m-patch14-384",
+    projector_layers=3,
+)
+```
+
+That's it. OpenLLaVA auto-detects hidden dimensions, builds the projector, and patches the tokenizer. No boilerplate. No config files.
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## Architecture
+
+</div>
+
+<br>
+
+<table>
+<tr>
+<td width="50%" valign="top">
+
+**Vision Encoder**
+
+Any encoder from HuggingFace — SigLIP 2, CLIP, EVA-CLIP, InternViT. OpenLLaVA auto-reads the output dimension and handles tokenization regardless of encoder architecture.
+
+<br>
+
+**Projector Engine**
+
+3-layer MLP with GELU activation, implemented as a fused CUDA kernel. Faster than PyTorch naive by design. Hidden dimension auto-computed from encoder output → LLM input.
+
+</td>
+<td width="50%" valign="top">
+
+**Model Patcher**
+
+Patches any HuggingFace causal LM to accept vision tokens. Adds `<image>` special token, extends the embedding layer, and wires the projector output into the LLM input stream. Supports LoRA-patched models.
+
+<br>
+
+**Training Engine**
+
+Two-phase training built in. Phase 1: projector warmup with frozen LLM. Phase 2: joint fine-tuning with LoRA. Gradient checkpointing, Flash Attention 2, and bfloat16 enabled by default.
+
+</td>
+</tr>
+</table>
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## Stack
+
+</div>
+
+<br>
+
+| Layer | Technology | Purpose |
+|:------|:----------:|:--------|
+| CUDA Kernels | C/CUDA | Fused projector ops, vision token attention |
+| Core | C++ | Memory management, tensor routing |
+| Bindings | pybind11 | C++ → Python bridge |
+| API | Python | Public interface |
+| Export | HuggingFace | Standard model format + GGUF |
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## Training Pipeline
+
+</div>
+
+<br>
+
+```python
+from openllava import OpenLLaVATrainer
+
+trainer = OpenLLaVATrainer(
+    model=model,
+    vision_encoder="google/siglip2-so400m-patch14-384",
+    pretrain_dataset="liuhaotian/LLaVA-Pretrain",   # Phase 1
+    instruct_dataset="liuhaotian/LLaVA-Instruct-150K",  # Phase 2
+    lora_r=64,
+    lora_alpha=128,
+    lora_target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
+)
+
+trainer.train()  # Handles both phases automatically
+```
+
+OpenLLaVA manages phase transitions, learning rate schedules, and checkpoint saving. You run one command.
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## OpceanAI Vision Models
+
+</div>
+
+<br>
+
+OpceanAI uses OpenLLaVA to publish vision versions of new language models as they release. These are the models built with the framework:
+
+<br>
+
+<table>
+<tr>
+<td width="50%" valign="top">
+
+**Yaki YuuKi+ Vision** *(in development)*
+
+Vision-language model built on Yuuki RxG 8B (DeepSeek-R1-Qwen2.5-8B fine-tune). Complex visual reasoning, bilingual (ES/EN), preserves the Yuuki `<think>` chain-of-thought behavior for multimodal tasks.
+
+Vision encoder: SigLIP 2 SO400M · LoRA r=64
+
+[![Status](https://img.shields.io/badge/Status-In_Development-orange?style=flat-square)](https://huggingface.co/OpceanAI)
+
+</td>
+<td width="50%" valign="top">
+
+**Yuuki NxG VL**
+
+7B vision-language model fine-tuned from Qwen2.5-VL-7B-Instruct. Extends the NxG model family to multimodal tasks. The first OpceanAI vision model and the validation case for the OpenLLaVA pipeline.
+
+[![Model](https://img.shields.io/badge/Yuuki_NxG_VL-HuggingFace-ffd21e?style=flat-square&logo=huggingface&logoColor=black)](https://huggingface.co/OpceanAI/Yuuki-NxG-vl)
+
+</td>
+</tr>
+</table>
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## Philosophy
+
+</div>
+
+<br>
+
+<table>
+<tr>
+<td width="50%" valign="top">
+
+**Model-Agnostic by Design**
+
+Every major framework for multimodal training — LLaVA, LLaVA-Next, InstructBLIP — is hardcoded to specific model families. OpenLLaVA is not. The projector adapts to any hidden dimension. The patcher works on any causal LM. The training engine handles any tokenizer.
+
+</td>
+<td width="50%" valign="top">
+
+**Speed Over Ceremony**
+
+When a new language model drops, the window to publish a vision version is 48–72 hours before the ecosystem moves on. OpenLLaVA is designed for that constraint — minimal configuration, automated phase management, one-command training.
+
+</td>
+</tr>
+</table>
+
+<table>
+<tr>
+<td width="50%" valign="top">
+
+**Low Level Where It Matters**
+
+The projector is the critical path. Everything else can be Python. The CUDA kernel for the fused MLP op and the C++ memory manager exist because training throughput on a single A100 is the binding constraint for a zero-budget lab.
+
+</td>
+<td width="50%" valign="top">
+
+**Fully Open**
+
+Apache 2.0. No gating. No commercial restrictions. The framework exists so that any researcher — with any model, any hardware, any budget — can build a competitive vision-language model.
+
+</td>
+</tr>
+</table>
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## Roadmap
+
+</div>
+
+<br>
+
+<table>
+<tr>
+<td width="50%" valign="top">
+
+**Framework**
+
+| Feature | Status |
+|:--------|:------:|
+| Python API + model patcher | In development |
+| MLP projector (PyTorch) | In development |
+| Two-phase training engine | In development |
+| Fused CUDA projector kernel | Planned |
+| C++ memory core | Planned |
+| GGUF vision export | Planned |
+| Multi-encoder support (BRAVE-style) | Planned |
+
+</td>
+<td width="50%" valign="top">
+
+**Vision Models**
+
+| Model | Status |
+|:------|:------:|
+| Yuuki NxG VL | Released |
+| Yaki YuuKi+ Vision (8B) | In development |
+| Community model pipeline | Planned |
+
+</td>
+</tr>
+</table>
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## Contributing
+
+</div>
+
+<br>
+
+OpenLLaVA is built to be extended. If you patch a model family that isn't supported yet, the contribution belongs in the framework. If you find a faster kernel implementation, open a PR.
+
+The project is maintained by OpceanAI but owned by the community.
+
+```bash
+git clone https://github.com/OpceanAI/openllava
+cd openllava
+pip install -e ".[dev]"
+```
+
+<br>
+
+---
+
+<br>
+
+<div align="center">
+
+## Built by OpceanAI
+
+</div>
+
+<br>
+
+<div align="center">
+
+OpenLLaVA is the vision infrastructure layer of [OpceanAI](https://huggingface.co/OpceanAI) — an independent AI research organization operating with no institutional funding, no cloud compute budget, and no team. Every model in the OpceanAI vision pipeline is trained on Google Colab Pro and validated on consumer hardware.
+
+<br>
+
+[![OpceanAI](https://img.shields.io/badge/OpceanAI-Research-0D1117?style=for-the-badge)](https://huggingface.co/OpceanAI)
+&nbsp;
+[![HuggingFace](https://img.shields.io/badge/Models-Hugging_Face-ffd21e?style=for-the-badge&logo=huggingface&logoColor=black)](https://huggingface.co/OpceanAI)
+&nbsp;
+[![Sponsor](https://img.shields.io/badge/Sponsor-GitHub_Sponsors-ea4aaa?style=for-the-badge&logo=githubsponsors&logoColor=white)](https://github.com/sponsors/aguitauwu)
+
+<br>
+
+---
+
+<br>
+
+**Open framework. Open models. Zero budget. Measurable results.**
+
+<br>
+
+[![OpenLLaVA](https://img.shields.io/badge/OpenLLaVA-2026-0D1117?style=for-the-badge)](https://github.com/OpceanAI/openllava)
+
+<br>
+
+*The fastest path from any language model to a vision-language model.*
+
+</div>