README.md

---
license: mit
tags:
- multimodal
- embeddings
datasets:
- ituperceptron/image-captioning-turkish
- dogukanvzr/ml-paraphrase-tr
library_name: pytorch
language:
- tr
base_model:
- newmindai/modernbert-base-tr-uncased-allnli-stsb
- facebook/dinov2-base
---

# Turkish Multimodal Embedding Model

This repository contains a **contrastively trained Turkish multimodal embedding model**, combining a text encoder and a vision encoder with projection heads.  
The model is trained entirely on **Turkish datasets** (image–caption and paraphrase), making it specifically tailored for Turkish multimodal applications.

## Model Summary
- **Text encoder**: `newmindai/modernbert-base-tr-uncased-allnli-stsb`
- **Vision encoder**: `facebook/dinov2-base`
- **Dimensions**: `text_dim=768`, `image_dim=768`, `embed_dim=768`
- **Projection dropout**: fixed at `0.4` (inside `ProjectionHead`)
- **Pooling**: mean pooling over tokens (`use_mean_pooling_for_text=True`)
- **Normalize outputs**: `{normalize}`
- **Encoders frozen during training?**: `{frozen}` (this release was trained with encoders **NOT frozen**)
- **Language focus**: Turkish (both text and image–caption pairs are fully in Turkish)

## Training Strategy (inspired by JINA-CLIP-v2 style)
- The model was trained jointly with **image–text** and **text–text** pairs using a **bidirectional contrastive loss** (InfoNCE/CLIP-style).
- For **image–text**, standard CLIP-style training with **in-batch negatives** was applied.
- For **text–text**, only **positive paraphrase pairs (label=1)** were used, with in-batch negatives coming from other samples.
- This follows the general training philosophy often seen in Jina’s multimodal work, but in a **simplified single-stage setup** (without the 3-stage curriculum).

## Datasets
- **Image–Text**: [`ituperceptron/image-captioning-turkish`](https://huggingface.co/datasets/ituperceptron/image-captioning-turkish)  
- **Text–Text (Paraphrase)**: [`dogukanvzr/ml-paraphrase-tr`](https://huggingface.co/datasets/dogukanvzr/ml-paraphrase-tr)  

> Both datasets are in Turkish, aligning the model’s embedding space around Turkish multimodal signals.  
> Please check each dataset’s license and terms before downstream use.

## Files
- `pytorch_model.bin` — PyTorch `state_dict`
- `config.json` — metadata (encoder IDs, dimensions, flags)
- `model.py` — custom model classes (required to load)
- (This README is the model card.)

## Evaluation Results
**Dataset:** Test split created from [`ituperceptron/image-captioning-turkish`](https://huggingface.co/datasets/ituperceptron/image-captioning-turkish)

### Image-Text
**Average cosine similarity:** 0.7934

**Recall@K**
<table>
<tr><th>Direction</th><th>R@1</th><th>R@5</th><th>R@10</th></tr>
<tr><td>Text → Image</td><td>0.9365</td><td>0.9913</td><td>0.9971</td></tr>
<tr><td>Image → Text</td><td>0.9356</td><td>0.9927</td><td>0.9958</td></tr>
</table>

<details>
<summary>Raw metrics (JSON)</summary>

```json
{
    "avg_cosine_sim": 0.7934404611587524,
    "recall_text_to_image": {
        "R@1": 0.936458564763386,
        "R@5": 0.9913352588313709,
        "R@10": 0.9971117529437903
    },
    "recall_image_to_text": {
        "R@1": 0.9355698733614752,
        "R@5": 0.9926682959342369,
        "R@10": 0.9957787158409243
    }
}
```
</details>

### Text-Text
**Average cosine similarity:** 0.7599

**Recall@K**
<table>
<tr><th>Direction</th><th>R@1</th><th>R@5</th><th>R@10</th></tr>
<tr><td>Text → Text</td><td>0.7198</td><td>0.9453</td><td>0.9824</td></tr>
</table>

<details>
<summary>Raw metrics (JSON)</summary>
  
```json
{
    "avg_cosine_sim": 0.7599335312843323,
    "recall_text_to_text": {
        "R@1": 0.719875500222321,
        "R@5": 0.9453090262338817,
        "R@10": 0.9824366385060027
    }
}
```
</details>

## Loading & Usage
```python
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer, AutoImageProcessor
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"

model_name = "utkubascakir/MultiEmbedTR"

model = AutoModel.from_pretrained(model_name, trust_remote_code=True).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
image_processor = AutoImageProcessor.from_pretrained(model_name, trust_remote_code=True)

model.eval()

# Text Embedding
texts = ["yeşil arka planlı bir kedi", "kumsalda bir köpek"]
text_inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    return_tensors="pt"
).to(device)

with torch.no_grad():
    text_embeds = model.encode_text(
        input_ids=text_inputs["input_ids"],
        attention_mask=text_inputs["attention_mask"]
    )  

print("Text embeddings shape:", text_embeds.shape)

# Image Embedding
img = Image.open("kedi.jpg").convert("RGB")
image_inputs = image_processor(
    images=img,
    return_tensors="pt"
).to(device)

with torch.no_grad():
    image_embeds = model.encode_image(
        pixel_values=image_inputs["pixel_values"]
    ) 

print("Image embeddings shape:", image_embeds.shape)

similarity = F.cosine_similarity(text_embeds, image_embeds)
print("Cosine similarity:", similarity)
```
  
## Limitations & Intended Use
This release provides a **Turkish multimodal embedding model**, trained to produce aligned vector representations for text and images.  
It has not been tested for specific downstream tasks (e.g., retrieval, classification).  
No guarantees for bias/toxicity; please evaluate on your own target domain.

## Citation
If you use this model, please cite this repository.