Update README.md
Browse files
README.md
CHANGED
|
@@ -1,6 +1,40 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
-
-
|
| 5 |
-
-
|
| 6 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
+
- Qwen/Qwen3-8B-Base
|
| 5 |
+
- DeepGlint-AI/rice-vit-large-patch14-560
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
# LLaVA-OneVision-1.5-8B Initialization Model Card
|
| 9 |
+
|
| 10 |
+
## ๐ Overview
|
| 11 |
+
|
| 12 |
+
This model provides an initialization checkpoint for training **LLaVA-OneVision-1.5**, designed to combine strong language and vision capabilities. It integrates a powerful LLM and a state-of-the-art vision encoder, with a flexible adapter to enable efficient multimodal learning.
|
| 13 |
+
|
| 14 |
+
## ๐๏ธ Key Components
|
| 15 |
+
|
| 16 |
+
- **Vision Encoder:**
|
| 17 |
+
Uses the pretrained ViT model from [DeepGlint-AI/rice-vit-large-patch14-560](https://huggingface.co/DeepGlint-AI/rice-vit-large-patch14-560) to extract rich visual features.
|
| 18 |
+
|
| 19 |
+
- **Adapter:**
|
| 20 |
+
A randomly initialized adapter module with 4ร token compression capability, enabling efficient fusion of image and text modalities.
|
| 21 |
+
|
| 22 |
+
- **Language Model:**
|
| 23 |
+
Incorporates the pretrained language model [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base) for robust text understanding and generation.
|
| 24 |
+
|
| 25 |
+
## ๐ Usage
|
| 26 |
+
|
| 27 |
+
This initialization checkpoint is intended for downstream training and fine-tuning. For usage and training scripts, please refer to the [EvolvingLMMs-Lab/LLaVA-OneVision-1.5 repository](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5).
|
| 28 |
+
|
| 29 |
+
## ๐ References
|
| 30 |
+
|
| 31 |
+
- [DeepGlint-AI/rice-vit-large-patch14-560](https://huggingface.co/DeepGlint-AI/rice-vit-large-patch14-560)
|
| 32 |
+
- [Qwen/Qwen3-8B-Base](https://huggingface.co/Qwen/Qwen3-8B-Base)
|
| 33 |
+
- [EvolvingLMMs-Lab/LLaVA-OneVision-1.5](https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5)
|
| 34 |
+
|
| 35 |
+
## โ๏ธ License
|
| 36 |
+
|
| 37 |
+
Apache 2.0
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
|