Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,81 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
base_model:
|
| 6 |
+
- OpenGVLab/InternVL2_5-38B
|
| 7 |
+
pipeline_tag: vision-language-model
|
| 8 |
+
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# SkyworkVL-38B: Multimodal Understanding with Bag of Tricks
|
| 12 |
+
|
| 13 |
+
<p align="center">
|
| 14 |
+
<img src="assets/skyworkvl_logo.png" alt="SkyworkVL Logo" width="60%">
|
| 15 |
+
</p>
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
<p align="center">
|
| 19 |
+
<a href="https://github.com/YourGitHub/SkyworkVL-38B" target="_blank">馃寪 Github</a> 路 馃憢 <a href="https://www.skyworkvl.ai" target="_blank">Playground</a> 路 馃挰 <a href="https://discord.gg/yourdiscord" target="_blank">Discord</a>
|
| 20 |
+
</p>
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
---
|
| 24 |
+
|
| 25 |
+
This repository contains Diffusers-format model weights for **SkyworkVL-38B**, an advanced vision-language model built on the base model [OpenGVLab/InternVL2_5-38B](https://huggingface.co/OpenGVLab/InternVL2_5-38B).
|
| 26 |
+
|
| 27 |
+
## Introduction
|
| 28 |
+
|
| 29 |
+
**SkyworkVL-38B** is a state-of-the-art VLM model trained on 2 million high-quality caption and QA data samples. Leveraging innovative techniques across multiple training stages, our model delivers superior performance on a range of vision-language tasks including multi-disciplinary question answering and scientific chart analysis.
|
| 30 |
+
|
| 31 |
+
## 馃攽 Key Features
|
| 32 |
+
|
| 33 |
+
### 1. Multi-Resolution Processing
|
| 34 |
+
|
| 35 |
+
- **Innovative Image Tiling:** Images are processed at multiple resolutions. For each resolution, we apply Closest Aspect Ratio Matching to partition the image into tiles. Finally, the original image is resized into a tile and appended to the final representation鈥攅nsuring comprehensive image understanding.
|
| 36 |
+
|
| 37 |
+
### 2. Multi-Stage Supervised Fine-Tuning (SFT)
|
| 38 |
+
|
| 39 |
+
- **Stage 1:** Fine-tuning on the full dataset.
|
| 40 |
+
- **Stage 2:** Refinement using a curated subset of 200K high-scoring samples filtered by GPT-4 evaluations.
|
| 41 |
+
- **Stage 3:** Further fine-tuning using mispredicted samples from Stage 2 alongside a small amount of generic domain data.
|
| 42 |
+
|
| 43 |
+
### 3. High-Quality Chain-of-Thought (CoT) Fine-Tuning
|
| 44 |
+
|
| 45 |
+
- **Enhanced Reasoning:** Integrates high-quality CoT data including self-collected multimodal Chinese Gaokao data with detailed analysis to boost the model鈥檚 reasoning capability.
|
| 46 |
+
|
| 47 |
+
### 4. GRPO + Rule-Based Reward Training
|
| 48 |
+
|
| 49 |
+
- **Performance Boost:** Utilizes GRPO and rule-based reward training to further refine output quality and overall performance.
|
| 50 |
+
|
| 51 |
+
## Model Introduction
|
| 52 |
+
|
| 53 |
+
| Model Name | Base Model | Parameters | Download Link |
|
| 54 |
+
| ----------------------- | ------------------------- | ---------- | ----------------------------------------------------------- |
|
| 55 |
+
| SkyworkVL-38B (Current) | OpenGVLab/InternVL2_5-38B | 38B | 馃 [Download](https://huggingface.co/YourRepo/SkyworkVL-38B) |
|
| 56 |
+
|
| 57 |
+
## Performance
|
| 58 |
+
|
| 59 |
+
| Metric | MathVista (testmini) | MMMU (val) | AI2D (BBox) | OCRBench | MME | **RealWorldQA** | **HallusionBench** |
|
| 60 |
+
| --------------------------- | -------------------- | --------------- | --------------- | ------------- | -------------- | --------------- | ------------------ |
|
| 61 |
+
| Internvl2.5-38B (瀹樻柟) | 71.9 | 63.9 | 87.6 | 842 | 2455 | 73.5 | 56.8 |
|
| 62 |
+
| **SkyworkVL-38B (Current)** | **74.4 (+2.5)** | **64.0 (+0.1)** | **88.4 (+0.8)** | **854 (+12)** | **2479 (+24)** | **76.9 (+3.4)** | **58.9 (+2.1)** |
|
| 63 |
+
|
| 64 |
+
*The performance improvements above demonstrate notable gains in multi-disciplinary question answering, object detection (BBox), and scientific chart analysis among other benchmarks.*
|
| 65 |
+
|
| 66 |
+
## Usage
|
| 67 |
+
|
| 68 |
+
Please refer to the [Guide](https://github.com/YourGitHub/SkyworkVL-38B) for detailed instructions on inference and integration.
|
| 69 |
+
|
| 70 |
+
## Citation
|
| 71 |
+
|
| 72 |
+
```BibTeX
|
| 73 |
+
@misc{SkyworkVL38B,
|
| 74 |
+
author = {Skywork-AI},
|
| 75 |
+
title = {SkyworkVL-38B: Multimodal Understanding with Bag of Tricks},
|
| 76 |
+
year = {2025},
|
| 77 |
+
publisher = {Huggingface},
|
| 78 |
+
journal = {Huggingface repository},
|
| 79 |
+
howpublished = {\url{https://huggingface.co/YourRepo/SkyworkVL-38B}}
|
| 80 |
+
}
|
| 81 |
+
```
|