---
license: apache-2.0
library_name: peft
tags:
  - sentence-transformers
  - siglip
  - lora
  - multimodal
  - email-clustering
  - astranexus
base_model:
  - sentence-transformers/all-MiniLM-L6-v2
  - google/siglip-large-patch16-384
---

# AstraNexus Multimodal Email Encoder (DriptoBhattacharyya/astranexus-mm-encoder)

LoRA-fine-tuned **late-fusion multimodal encoder** for unsupervised email
clustering. Text branch = MiniLM (`all-MiniLM-L6-v2`), vision branch =
`google/siglip-large-patch16-384`, joined by a learned
256-d fusion head. Trained with **supervised contrastive loss** on
600 teacher-labeled emails (topic labels distilled from
Qwen2.5-7B-Instruct), 6 epochs on 2×T4.

## Why this exists
Off-the-shelf fixed-weight fusion can't win both text-clear and image-decisive
emails (no single α is best for both). This encoder *learns* the fusion, so it
clusters hard cases (generic subject + topic-revealing attachment) correctly.

## Results — independent hard eval (120 emails, image-decisive)
| | ARI | NMI |
|---|---|---|
| Off-the-shelf fused (gate) | 0.168 | 0.519 |
| **Fine-tuned (this model)** | **0.252** | **0.511** |

Δ ARI **+0.084**. The gate is an honest off-the-shelf
baseline on the *same* encoders; the lift is purely from the LoRA + learned fusion.

## Files
- `text_lora/` — PEFT-LoRA adapter for the MiniLM text tower
- `vision_lora/` — PEFT-LoRA adapter for the SigLIP vision tower
- `proj.pt` — learned fusion head (`Linear(t+v, 256) -> GELU -> Linear(256, 256)`)

## Usage
See `astranexus/cluster/ft_encoder.py` in the AstraNexus repo — loads both
adapters + the fusion head and exposes `encode(emails) -> np.ndarray`.

## Reproducibility note
Trained on Kaggle (torch 2.10, 2×T4). The SigLIP LoRA adapter keys use the
`vision_model.` module layout from the training-time `transformers`; newer
`transformers` (5.x) flattened SigLIP, which shifts both the adapter key paths
**and** the frozen base-model numerics. `ft_encoder._load_adapter_robust`
remaps the keys, but for faithful results pin `transformers` to the training
line (4.x) and install `torchvision` (matches the image processor). The eval
numbers above were measured in the training environment.