| # Daily Update — Embedding Fusion Experiments |
| *Image embedding benchmark · Tobacco-3482* |
|
|
| --- |
|
|
| ## Completed |
|
|
| - Ran embedding fusion experiments across 7 model pairs, testing whether combining embeddings from two complementary models produces better representations than either model alone. |
| - Each pair was tested with 5 variants: raw concat at equal weight (α=0.5), PCA-whitened concat, and an alpha sweep at α=0.6, 0.7, and 0.8 to up-weight the stronger model. |
| - Results saved to `results/fusion_summary.txt` and `results/fusion_summary.json`. |
| - Updated `README.md` to include the full fusion section with all 7 pairs, strategies, and a link to the result files. |
|
|
| --- |
|
|
| ## In Progress |
|
|
| - Analysing fusion results to identify which pair and strategy performs best across clf accuracy and kNN retrieval. |
| - Investigating whether fusion helps ModernVBERT close the gap with SigLIP. |
|
|
| --- |
|
|
| ## Blockers |
|
|
| - None this session. |
|
|
| --- |
|
|
| ## Key Learnings |
|
|
| - **What fusion is:** Embedding fusion combines the vector representations from two separate models into a single longer vector. The intuition is that different models pick up on different visual features — for example SigLIP learns vision-language alignment while DiT learns document-specific layout and texture patterns from 42 million scanned documents. Concatenating their embeddings gives the downstream classifier access to both types of signal at once. |
|
|
| - **Why weighted concatenation:** Rather than treating both models equally, a weighted concat (α × emb_A, (1−α) × emb_B) lets you control how much each model contributes. At α=0.5 both are equal. Sweeping α ∈ {0.6, 0.7, 0.8} tests whether leaning more on the stronger model (SigLIP2 SO400M in this case) improves results, since the two models are unlikely to be equally useful for this task. |
|
|
| - **Why PCA-whitened concat:** Simply concatenating two embedding spaces can introduce redundancy — both models may encode similar low-level features (edges, patches), so the combined vector wastes dimensions on duplicate information. PCA whitening after concatenation removes correlations across the joint space and rescales each dimension to unit variance, producing a more compact and uniform representation. This is the same post-processing approach that helped single-model clf accuracy earlier in the benchmark. |
|
|
| - **Fusion pair rationale:** The 7 pairs were chosen to test complementary pretraining signals. SigLIP/SigLIP2 bring vision-language alignment trained on image-text pairs. DiT brings document-specific pretraining on IIT-CDIP scanned documents. DINOv2 brings self-supervised vision features with no label supervision at all. Pairing a VLM with a doc-specific or vision-only model tests whether the missing signal in one model is captured by the other. |
|
|
| - **Results reference:** Full results in [`results/fusion_summary.txt`](results/fusion_summary.txt) and [`results/fusion_summary.json`](results/fusion_summary.json). |
