jiangbop commited on
Commit
792b8d3
verified
1 Parent(s): 76abcf7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +81 -3
README.md CHANGED
@@ -1,3 +1,81 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ base_model:
6
+ - OpenGVLab/InternVL2_5-38B
7
+ pipeline_tag: vision-language-model
8
+
9
+ ---
10
+
11
+ # SkyworkVL-38B: Multimodal Understanding with Bag of Tricks
12
+
13
+ <p align="center">
14
+ <img src="assets/skyworkvl_logo.png" alt="SkyworkVL Logo" width="60%">
15
+ </p>
16
+
17
+
18
+ <p align="center">
19
+ <a href="https://github.com/YourGitHub/SkyworkVL-38B" target="_blank">馃寪 Github</a> 路 馃憢 <a href="https://www.skyworkvl.ai" target="_blank">Playground</a> 路 馃挰 <a href="https://discord.gg/yourdiscord" target="_blank">Discord</a>
20
+ </p>
21
+
22
+
23
+ ---
24
+
25
+ This repository contains Diffusers-format model weights for **SkyworkVL-38B**, an advanced vision-language model built on the base model [OpenGVLab/InternVL2_5-38B](https://huggingface.co/OpenGVLab/InternVL2_5-38B).
26
+
27
+ ## Introduction
28
+
29
+ **SkyworkVL-38B** is a state-of-the-art VLM model trained on 2 million high-quality caption and QA data samples. Leveraging innovative techniques across multiple training stages, our model delivers superior performance on a range of vision-language tasks including multi-disciplinary question answering and scientific chart analysis.
30
+
31
+ ## 馃攽 Key Features
32
+
33
+ ### 1. Multi-Resolution Processing
34
+
35
+ - **Innovative Image Tiling:** Images are processed at multiple resolutions. For each resolution, we apply Closest Aspect Ratio Matching to partition the image into tiles. Finally, the original image is resized into a tile and appended to the final representation鈥攅nsuring comprehensive image understanding.
36
+
37
+ ### 2. Multi-Stage Supervised Fine-Tuning (SFT)
38
+
39
+ - **Stage 1:** Fine-tuning on the full dataset.
40
+ - **Stage 2:** Refinement using a curated subset of 200K high-scoring samples filtered by GPT-4 evaluations.
41
+ - **Stage 3:** Further fine-tuning using mispredicted samples from Stage 2 alongside a small amount of generic domain data.
42
+
43
+ ### 3. High-Quality Chain-of-Thought (CoT) Fine-Tuning
44
+
45
+ - **Enhanced Reasoning:** Integrates high-quality CoT data including self-collected multimodal Chinese Gaokao data with detailed analysis to boost the model鈥檚 reasoning capability.
46
+
47
+ ### 4. GRPO + Rule-Based Reward Training
48
+
49
+ - **Performance Boost:** Utilizes GRPO and rule-based reward training to further refine output quality and overall performance.
50
+
51
+ ## Model Introduction
52
+
53
+ | Model Name | Base Model | Parameters | Download Link |
54
+ | ----------------------- | ------------------------- | ---------- | ----------------------------------------------------------- |
55
+ | SkyworkVL-38B (Current) | OpenGVLab/InternVL2_5-38B | 38B | 馃 [Download](https://huggingface.co/YourRepo/SkyworkVL-38B) |
56
+
57
+ ## Performance
58
+
59
+ | Metric | MathVista (testmini) | MMMU (val) | AI2D (BBox) | OCRBench | MME | **RealWorldQA** | **HallusionBench** |
60
+ | --------------------------- | -------------------- | --------------- | --------------- | ------------- | -------------- | --------------- | ------------------ |
61
+ | Internvl2.5-38B (瀹樻柟) | 71.9 | 63.9 | 87.6 | 842 | 2455 | 73.5 | 56.8 |
62
+ | **SkyworkVL-38B (Current)** | **74.4 (+2.5)** | **64.0 (+0.1)** | **88.4 (+0.8)** | **854 (+12)** | **2479 (+24)** | **76.9 (+3.4)** | **58.9 (+2.1)** |
63
+
64
+ *The performance improvements above demonstrate notable gains in multi-disciplinary question answering, object detection (BBox), and scientific chart analysis among other benchmarks.*
65
+
66
+ ## Usage
67
+
68
+ Please refer to the [Guide](https://github.com/YourGitHub/SkyworkVL-38B) for detailed instructions on inference and integration.
69
+
70
+ ## Citation
71
+
72
+ ```BibTeX
73
+ @misc{SkyworkVL38B,
74
+ author = {Skywork-AI},
75
+ title = {SkyworkVL-38B: Multimodal Understanding with Bag of Tricks},
76
+ year = {2025},
77
+ publisher = {Huggingface},
78
+ journal = {Huggingface repository},
79
+ howpublished = {\url{https://huggingface.co/YourRepo/SkyworkVL-38B}}
80
+ }
81
+ ```