README.md

metadata

license: apache-2.0
language:
  - en
  - zh
base_model:
  - Qwen/Qwen3-4B-Instruct-2507
  - Qwen/Qwen3-VL-4B-Instruct
pipeline_tag: image-text-to-text
tags:
  - merge

This is an experimental model. While retaining its visual capabilities, we aim to align its text performance with that of pure text models.

Model Highlights:

• merge method: ASDF

• Highest precision: dtype: float32 + out_dtype: bfloat16

• Context length: 262,144

Parameter Settings:

Temperature=0.7, TopP=0.8, TopK=20,MinP=0.

Why Can Models Be Merged:

• The tensors for the text portion are exactly the same between the visual model and the pure text model.
• In terms of tensor naming, the only difference between vision models and pure text models is the addition of ".language_model".
• Therefore, by uniformly removing this part before merging, the text-related tensors of the two can be directly merged.

How Exactly Are Models Merged:

Input

Given two weight tensors from models with identical architecture (Text and Vision branches):
$$T^{\text{text}}\in {\mathbb{R}}^{d_1\times \cdots \times d_n},T^{\text{vision}}\in {\mathbb{R}}^{d_1\times \cdots \times d_n}T^\{\backslash text\{text\}\}\; \backslash in\; \backslash mathbb\{R\}^\{d\_1\; \backslash times\; \backslash cdots\; \backslash times\; d\_n\},\; \backslash quad\; T^\{\backslash text\{vision\}\}\; \backslash in\; \backslash mathbb\{R\}^\{d\_1\; \backslash times\; \backslash cdots\; \backslash times\; d\_n\}$$
For each vision tensor key k_v, strip the "language_model." prefix to obtain the corresponding text model key for matching.

---

Step 1: Special Layer Filtering

Skip merging for embedding and language modeling head layers:

• If tensor name contains "embed" or "lm_head", return T^vision directly.
• Proceed only if both tensors have the same shape.

---

Step 2: Type Conversion and Delta Computation

Convert to float32 for numerical stability and compute the difference tensor:
$$W^{\text{text}}=T^{\text{text}}\mathrm{.}\text{float}(),W^{\text{vision}}=T^{\text{vision}}\mathrm{.}\text{float}()W^\{\backslash text\{text\}\}\; =\; T^\{\backslash text\{text\}\}.\backslash text\{float\}(),\; \backslash quad\; W^\{\backslash text\{vision\}\}\; =\; T^\{\backslash text\{vision\}\}.\backslash text\{float\}()$$
$$\mathrm{\Delta }=W^{\text{vision}}-W^{\text{text}}\backslash Delta\; =\; W^\{\backslash text\{vision\}\}\; -\; W^\{\backslash text\{text\}\}$$

---

Step 3: Early Exit for Low-Rank Tensors

If Delta is a vector (i.e., rank < 2, such as bias or LayerNorm parameters), return T^vision directly.

---

Step 4: SVD Decomposition of Delta

Perform thin SVD on the difference tensor:
$$\mathrm{\Delta }=U\mathrm{\Sigma }{V}^{\mathrm{\top }},U\in {\mathbb{R}}^{m\times r},\mathrm{\Sigma }\in {\mathbb{R}}^{r\times r},V\in {\mathbb{R}}^{n\times r}\backslash Delta\; =\; U\; \backslash Sigma\; V^\backslash top,\; \backslash quad\; U\; \backslash in\; \backslash mathbb\{R\}^\{m\; \backslash times\; r\},\backslash \; \backslash Sigma\; \backslash in\; \backslash mathbb\{R\}^\{r\; \backslash times\; r\},\backslash \; V\; \backslash in\; \backslash mathbb\{R\}^\{n\; \backslash times\; r\}$$
where r = min(m, n), and Σ = diag(σ₁, …, σᵣ) with σ₁ ≥ ⋯ ≥ σᵣ ≥ 0.

---

Step 5: Automatic Rank Selection via Knee Point Detection

5.1 Normalize singular values and indices

Let s = (σ₁, …, σᵣ). Normalize to unit square:
$$x_i=\frac{i-1}{r-1},y_i=\frac{{\sigma }_i-{\sigma }_r}{{\sigma }_1-{\sigma }_r+\epsilon },i=1,\dots ,rx\_i\; =\; \backslash frac\{i\; -\; 1\}\{r\; -\; 1\},\; \backslash quad\; y\_i\; =\; \backslash frac\{\backslash sigma\_i\; -\; \backslash sigma\_r\}\{\backslash sigma\_1\; -\; \backslash sigma\_r\; +\; \backslash varepsilon\},\; \backslash quad\; i\; =\; 1,\backslash dots,r$$

5.2 Compute perpendicular distance to line from first to last point

Line from (0, y_1) to (1, y_r) has direction vector (1, y_r - y_1).
For each point (x_i, y_i), compute normalized cross-product distance:
$$d_i=\mid (x_i)(y_r-y_1)-(y_i-y_1)(1)\mid d\_i\; =\; \backslash left|\; (x\_i)(y\_r\; -\; y\_1)\; -\; (y\_i\; -\; y\_1)(1)\; \backslash right|$$

5.3 Select knee index

$$k=\arg \underset{i}{\max }{d}_i,k=\max (1,k)k\; =\; \backslash arg\backslash max\_i\; d\_i,\; \backslash quad\; k\; =\; \backslash max(1,\; k)$$

---

Step 6: Low-Rank Reconstruction of Delta

Reconstruct Delta using top-k components:
$${\mathrm{\Delta }}_{\text{clean}}=U[:,:k]\cdot \mathrm{diag}({\sigma }_1,\dots ,{\sigma }_k)\cdot V^{\mathrm{\top }}[:k,:]\backslash Delta\_\{\backslash text\{clean\}\}\; =\; U[:,\; :k]\; \backslash cdot\; \backslash operatorname\{diag\}(\backslash sigma\_1,\; \backslash dots,\; \backslash sigma\_k)\; \backslash cdot\; V^\backslash top[:k,\; :]$$

---

Step 7: Fuse into Final Tensor

Add cleaned delta to text base:
$$W^{\text{merged}}=W^{\text{text}}+{\mathrm{\Delta }}_{\text{clean}}W^\{\backslash text\{merged\}\}\; =\; W^\{\backslash text\{text\}\}\; +\; \backslash Delta\_\{\backslash text\{clean\}\}$$
Cast back to original dtype (e.g., bfloat16):
$$\widehat{T}=W^{\text{merged}}\mathrm{.}\text{to}(T^{\text{text}}\mathrm{.}\text{dtype})\backslash hat\{T\}\; =\; W^\{\backslash text\{merged\}\}.\backslash text\{to\}(T^\{\backslash text\{text\}\}.\backslash text\{dtype\})$$

At the End:

• This merging algorithm is based on the following assumption: the visual capability of the model is mainly concentrated in a few larger singular values within the residual terms.
• It should be noted that we have not yet conducted a systematic evaluation of the model's visual capabilities, and this is only used here to demonstrate the feasibility of the merging technique.
• At the same time, we call for further research into model merging methods that unify vision and text, in order to find truly suitable merging algorithms.