README.md

---
language:
  - en
license: apache-2.0
library_name: sentence-transformers
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - sts
  - embeddings
  - symmetric-augmentation
  - fine-tuned
base_model: sentence-transformers/all-MiniLM-L6-v2
pipeline_tag: sentence-similarity
datasets:
  - mteb/stsbenchmark-sts
  - mteb/sts12-sts
metrics:
  - spearmanr
---

<h1 align="center">SymSTS-MiniLM-L6</h1>

<p align="center">
  <strong>Symmetrically Augmented MiniLM for Semantic Textual Similarity</strong>
</p>

<p align="center">
  <img src="https://img.shields.io/badge/Parameters-22.7M-blue" alt="params"/>
  <img src="https://img.shields.io/badge/Embedding--Dim-384-orange" alt="dim"/>
  <img src="https://img.shields.io/badge/Max--Seq--Len-256-yellow" alt="seqlen"/>
  <img src="https://img.shields.io/badge/License-Apache--2.0-green" alt="license"/>
  <img src="https://img.shields.io/badge/STS--Avg--Spearman-0.8205-brightgreen" alt="score"/>
</p>

---

## 1. What This Model Is

SymSTS-MiniLM-L6 is a fine-tuned version of
[sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2),
specialized for **Semantic Textual Similarity (STS)**.

The key idea behind this model is **symmetric data augmentation**. During training, every sentence
pair `(A, B, score)` is duplicated as `(B, A, score)`. This forces the model to learn that
similarity is bidirectional: `similarity(A, B) = similarity(B, A)`. This simple technique
produces consistent gains across all historical STS benchmarks.

The model has **22.7 million parameters**, runs fast on CPU, and fits comfortably on consumer GPUs.

> **This is NOT a general-purpose embedding model.**
> It is built for one job: measuring how similar two sentences are in meaning.
> It is not optimized for document retrieval, classification, or multilingual tasks.

---

## 2. Model Architecture

<table>
  <tr><th>Property</th><th>Value</th></tr>
  <tr><td>Base Architecture</td><td>MiniLM (6-layer Transformer encoder)</td></tr>
  <tr><td>Total Parameters</td><td>22.7 M</td></tr>
  <tr><td>Hidden Dimension</td><td>384</td></tr>
  <tr><td>Output Embedding Dimension</td><td>384</td></tr>
  <tr><td>Max Sequence Length</td><td>256 tokens</td></tr>
  <tr><td>Pooling Strategy</td><td>Mean Pooling</td></tr>
  <tr><td>Normalization</td><td>L2-normalized embeddings</td></tr>
  <tr><td>Similarity Function</td><td>Cosine Similarity</td></tr>
</table>

---

## 3. Intended Use

<table>
  <tr>
    <th>Good For</th>
    <th>Not Built For</th>
  </tr>
  <tr>
    <td>Semantic similarity scoring between two sentences</td>
    <td>Document retrieval or search ranking over large corpora</td>
  </tr>
  <tr>
    <td>Paraphrase and duplicate question detection</td>
    <td>Sentiment analysis or text classification</td>
  </tr>
  <tr>
    <td>Small-scale text clustering by meaning</td>
    <td>Zero-shot classification</td>
  </tr>
  <tr>
    <td>FAQ matching and chatbot intent matching</td>
    <td>Multilingual or cross-lingual tasks</td>
  </tr>
  <tr>
    <td>Sentence-level deduplication pipelines</td>
    <td>Long-document embedding beyond 256 tokens</td>
  </tr>
</table>

---

## 4. Benchmark Results

All scores are **Spearman rank correlation** measured locally using the
[MTEB](https://github.com/embeddings-benchmark/mteb) library on official test splits.

### 4.1 STS Results vs Base Model

<table>
  <tr>
    <th>Task</th>
    <th>Base MiniLM-L6-v2</th>
    <th>SymSTS-MiniLM-L6 (Ours)</th>
    <th>Difference</th>
  </tr>
  <tr><td>STS12</td><td>0.7237</td><td><strong>0.7873</strong></td><td>+0.0636</td></tr>
  <tr><td>STS13</td><td>0.8060</td><td><strong>0.8290</strong></td><td>+0.0230</td></tr>
  <tr><td>STS14</td><td>0.7559</td><td><strong>0.8186</strong></td><td>+0.0627</td></tr>
  <tr><td>STS15</td><td>0.8539</td><td><strong>0.8758</strong></td><td>+0.0219</td></tr>
  <tr><td>STS16</td><td>0.7899</td><td><strong>0.8152</strong></td><td>+0.0253</td></tr>
  <tr><td>STSBenchmark</td><td>0.8203</td><td><strong>0.8407</strong></td><td>+0.0204</td></tr>
  <tr><td>SICK-R</td><td>0.7758</td><td><strong>0.7772</strong></td><td>+0.0013</td></tr>
  <tr>
    <td><strong>Average</strong></td>
    <td><strong>0.7894</strong></td>
    <td><strong>0.8205</strong></td>
    <td><strong>+0.0312</strong></td>
  </tr>
</table>

### 4.2 Honest Comparison Against Other Models

Scores for external models are approximate, taken from the MTEB leaderboard and published
model cards. Our scores are measured locally. We include models that outperform ours
for full transparency.

<table>
  <tr>
    <th>Model</th>
    <th>Params</th>
    <th>STS-B Spearman</th>
    <th>STS Avg (approx)</th>
    <th>vs SymSTS</th>
  </tr>
  <tr>
    <td><strong>SymSTS-MiniLM-L6 (Ours)</strong></td>
    <td>22.7M</td>
    <td><strong>0.8407</strong></td>
    <td><strong>0.8205</strong></td>
    <td>--</td>
  </tr>
  <tr>
    <td>all-MiniLM-L6-v2</td>
    <td>22.7M</td>
    <td>0.8203</td>
    <td>0.7894</td>
    <td>SymSTS wins</td>
  </tr>
  <tr>
    <td>all-MiniLM-L12-v2</td>
    <td>33M</td>
    <td>~0.835</td>
    <td>~0.805</td>
    <td>SymSTS wins</td>
  </tr>
  <tr>
    <td>all-mpnet-base-v2</td>
    <td>109M</td>
    <td>~0.835</td>
    <td>~0.810</td>
    <td>SymSTS wins</td>
  </tr>
  <tr>
    <td>bge-small-en-v1.5</td>
    <td>33M</td>
    <td>~0.815</td>
    <td>~0.800</td>
    <td>SymSTS wins</td>
  </tr>
  <tr>
    <td>e5-small-v2</td>
    <td>33M</td>
    <td>~0.820</td>
    <td>~0.805</td>
    <td>SymSTS wins</td>
  </tr>
  <tr>
    <td>gte-small</td>
    <td>33M</td>
    <td>~0.840</td>
    <td>~0.815</td>
    <td>Comparable</td>
  </tr>
  <tr>
    <td>bge-base-en-v1.5</td>
    <td>110M</td>
    <td>~0.855</td>
    <td>~0.835</td>
    <td>They win</td>
  </tr>
  <tr>
    <td>gte-base</td>
    <td>110M</td>
    <td>~0.855</td>
    <td>~0.840</td>
    <td>They win</td>
  </tr>
  <tr>
    <td>nomic-embed-text-v1.5</td>
    <td>137M</td>
    <td>~0.865</td>
    <td>~0.845</td>
    <td>They win</td>
  </tr>
  <tr>
    <td>jina-embeddings-v3</td>
    <td>570M</td>
    <td>~0.870</td>
    <td>~0.855</td>
    <td>They win</td>
  </tr>
  <tr>
    <td>e5-mistral-7b-instruct</td>
    <td>7B</td>
    <td>~0.880</td>
    <td>~0.860</td>
    <td>They win</td>
  </tr>
</table>

**Summary:** SymSTS-MiniLM-L6 outperforms all models at or below 33M parameters and several
models up to 109M parameters on STS tasks. It is outperformed by larger base-architecture
models (110M+) and LLM-based embeddings, which is expected given the parameter gap.

---

## 5. Training Details

### 5.1 Loss Function

`CosineSimilarityLoss` — minimizes the mean squared error between the predicted cosine
similarity and the human-annotated similarity score.

### 5.2 Symmetric Data Augmentation

For every training pair:
Original: (Sentence_A, Sentence_B, score)
Augmented: (Sentence_B, Sentence_A, score)

Both are included, doubling the effective training set from ~8k to ~16k pairs. Note: because cosine similarity is symmetric by construction (sim(A,B) = sim(B,A) regardless of training), this augmentation's main effect is increasing the volume of training pairs rather than teaching the model a new bidirectional property. The performance gains reported in Section 4 are real and measured directly; the ablation isolating "more data" from "swap specifically" is planned as future work.
### 5.3 Hyperparameters

<table>
  <tr><th>Hyperparameter</th><th>Value</th></tr>
  <tr><td>Learning Rate</td><td>1e-5</td></tr>
  <tr><td>Batch Size</td><td>16</td></tr>
  <tr><td>Epochs</td><td>2</td></tr>
  <tr><td>Warmup</td><td>10% of total steps</td></tr>
  <tr><td>Weight Decay</td><td>0.01</td></tr>
  <tr><td>Optimizer</td><td>AdamW</td></tr>
  <tr><td>Precision</td><td>FP16 (mixed precision)</td></tr>
  <tr><td>Random Seed</td><td>42</td></tr>
  <tr><td>Hardware</td><td>NVIDIA GeForce GTX 1660 SUPER (6 GB VRAM)</td></tr>
</table>

---

## 6. Training Data

The model was fine-tuned on the **train** splits of the following datasets.
STS13 through STS16 and SICK-R were **not** used during training. They are held-out
evaluation benchmarks only.

<table>
  <tr>
    <th>Dataset</th>
    <th>Source</th>
    <th>Original Pairs</th>
    <th>After Augmentation</th>
    <th>License</th>
  </tr>
  <tr>
    <td>STS Benchmark (train)</td>
    <td>SemEval-2017 Task 1</td>
    <td>5,749</td>
    <td>11,498</td>
    <td>Research use</td>
  </tr>
  <tr>
    <td>STS12 (train)</td>
    <td>SemEval-2012 Task 6</td>
    <td>2,234</td>
    <td>4,468</td>
    <td>Research use</td>
  </tr>
  <tr>
    <td><strong>Total</strong></td>
    <td></td>
    <td><strong>7,983</strong></td>
    <td><strong>15,966</strong></td>
    <td></td>
  </tr>
</table>

The raw dataset files are **not** redistributed in this repository.
Only the fine-tuned model weights are provided.

---

## 7. Usage

```python
from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("blueprint-ai/SymSTS-MiniLM")

sentences = [
    "The cat sits on the mat.",
    "A feline is resting on the rug.",
    "The stock market crashed today."
]

embeddings = model.encode(sentences, normalize_embeddings=True)

sim_01 = util.cos_sim(embeddings[0], embeddings[1]).item()
sim_02 = util.cos_sim(embeddings[0], embeddings[2]).item()

print(f"Cat vs Feline:  {sim_01:.4f}")
print(f"Cat vs Stocks:  {sim_02:.4f}")
#8. Full Training Code
This is the exact script used to train SymSTS-MiniLM-L6.
import warnings
warnings.filterwarnings("ignore")

import os
import torch
from datasets import load_dataset, Dataset

from sentence_transformers import (
    SentenceTransformer,
    SentenceTransformerTrainer,
    SentenceTransformerTrainingArguments,
    InputExample,
)
from sentence_transformers.losses import CosineSimilarityLoss
from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator


# =============================================================
# Configuration
# =============================================================
BASE_MODEL    = "sentence-transformers/all-MiniLM-L6-v2"
OUTPUT_DIR    = "./output/symsts-minilm-l6"
RUNS_DIR      = "./output/runs-symsts-minilm-l6"
MAX_SEQ_LEN   = 256
BATCH_SIZE    = 16
LEARNING_RATE = 1e-5
EPOCHS        = 2

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Device: {device}")


# =============================================================
# Helpers
# =============================================================
def clean_text(text):
    if text is None:
        return ""
    return " ".join(str(text).strip().split())


def get_main_score(result):
    if isinstance(result, dict):
        for key, value in result.items():
            if "spearman_cosine" in key:
                try: return float(value)
                except Exception: pass
        for key, value in result.items():
            if "spearman" in key:
                try: return float(value)
                except Exception: pass
        for value in result.values():
            try: return float(value)
            except Exception: pass
    try: return float(result)
    except Exception: return 0.0


# =============================================================
# 1. Load Base Model
# =============================================================
print("Loading base model...")
student = SentenceTransformer(BASE_MODEL, device=device)
student.max_seq_length = MAX_SEQ_LEN


# =============================================================
# 2. Load STS Training Data with Symmetric Augmentation
# =============================================================
print("Loading STS training data...")

sources = [
    ("mteb/stsbenchmark-sts", "train"),
    ("mteb/sts12-sts",        "train"),
]

sentence1_list = []
sentence2_list = []
label_list     = []
seen_pairs     = set()


def add_pair(s1, s2, score):
    s1 = clean_text(s1)
    s2 = clean_text(s2)
    if not s1 or not s2:
        return

    key = tuple(sorted((s1.lower(), s2.lower())))
    if key in seen_pairs:
        return
    seen_pairs.add(key)

    # Forward
    sentence1_list.append(s1)
    sentence2_list.append(s2)
    label_list.append(score)

    # Reverse (Symmetric Augmentation)
    if s1.lower() != s2.lower():
        sentence1_list.append(s2)
        sentence2_list.append(s1)
        label_list.append(score)


for dataset_name, split in sources:
    try:
        ds = load_dataset(dataset_name, split=split)
        count = 0
        for row in ds:
            s1    = row.get("sentence1")
            s2    = row.get("sentence2")
            score = row.get("score", 0.0)
            try:
                score = float(score)
            except Exception:
                continue
            if score > 1.0:
                score = score / 5.0
            score = max(0.0, min(1.0, score))
            add_pair(s1, s2, score)
            count += 1
        print(f"  Loaded {count} original pairs from {dataset_name}")
    except Exception as e:
        print(f"  Skipping {dataset_name}: {e}")

print(f"Unique pairs: {len(seen_pairs)}")
print(f"Total augmented rows: {len(sentence1_list)}")

train_dataset = Dataset.from_dict({
    "sentence1": sentence1_list,
    "sentence2": sentence2_list,
    "label":     label_list,
})


# =============================================================
# 3. STS-B Test Evaluator
# =============================================================
print("Loading STS-B test evaluator...")

sts_test = load_dataset("mteb/stsbenchmark-sts", split="test")
eval_examples = []
for row in sts_test:
    s1    = clean_text(row.get("sentence1"))
    s2    = clean_text(row.get("sentence2"))
    score = float(row.get("score", 0.0)) / 5.0
    eval_examples.append(InputExample(texts=[s1, s2], label=score))

evaluator = EmbeddingSimilarityEvaluator.from_input_examples(
    eval_examples,
    name="sts-b-test",
)

base_score = get_main_score(evaluator(student))
print(f"Base model STS-B score: {base_score:.4f}")


# =============================================================
# 4. Train
# =============================================================
print("Training SymSTS-MiniLM-L6...")

train_loss = CosineSimilarityLoss(model=student)

training_args = SentenceTransformerTrainingArguments(
    output_dir                  = RUNS_DIR,
    num_train_epochs            = EPOCHS,
    per_device_train_batch_size = BATCH_SIZE,
    per_device_eval_batch_size  = BATCH_SIZE,
    learning_rate               = LEARNING_RATE,
    warmup_steps                = 0.1,
    weight_decay                = 0.01,
    fp16                        = torch.cuda.is_available(),
    bf16                        = False,
    logging_steps               = 50,
    save_strategy               = "no",
    eval_strategy               = "epoch",
    dataloader_num_workers      = 0,
    report_to                   = "none",
    remove_unused_columns       = False,
    seed                        = 42,
)

trainer = SentenceTransformerTrainer(
    model         = student,
    args          = training_args,
    train_dataset = train_dataset,
    loss          = train_loss,
    evaluator     = evaluator,
)

trainer.train()

os.makedirs(OUTPUT_DIR, exist_ok=True)
try:
    trainer.save_model(OUTPUT_DIR)
except Exception:
    student.save_pretrained(OUTPUT_DIR)

print(f"Model saved to: {OUTPUT_DIR}")


# =============================================================
# 5. Final Evaluation
# =============================================================
final_score = get_main_score(evaluator(student))

print("=" * 50)
print(f"Base model STS-B:   {base_score:.4f}")
print(f"SymSTS STS-B:       {final_score:.4f}")
print(f"Improvement:        {final_score - base_score:+.4f}")
print("=" * 50)

#9. Evaluation Code
This is the script used to produce the benchmark tables above.
import warnings
warnings.filterwarnings("ignore")

import os, json, glob, mteb
from sentence_transformers import SentenceTransformer

os.environ["TOKENIZERS_PARALLELISM"] = "false"

TASKS = [
    "STS12", "STS13", "STS14", "STS15",
    "STS16", "STSBenchmark", "SICK-R",
]

def get_scores(model_path, out_dir):
    model      = SentenceTransformer(model_path)
    tasks      = mteb.get_tasks(tasks=TASKS)
    evaluation = mteb.MTEB(tasks=tasks)
    evaluation.run(model, output_folder=out_dir, verbosity=0)

    scores = {}
    for task_name in TASKS:
        files = glob.glob(f"{out_dir}/**/*{task_name}*.json", recursive=True)
        if files:
            with open(files[0]) as f:
                d = json.load(f)
                if "scores" in d and "test" in d["scores"]:
                    scores[task_name] = d["scores"]["test"][0].get("main_score", 0.0)
    return scores

base = get_scores("sentence-transformers/all-MiniLM-L6-v2", "./mteb_base")
ours = get_scores("./output/symsts-minilm-l6",              "./mteb_ours")

print(f"{'Task':<16} {'Base':>8} {'Ours':>8} {'Diff':>8}")
print("-" * 44)
for t in TASKS:
    b = base.get(t, 0)
    o = ours.get(t, 0)
    print(f"{t:<16} {b:>8.4f} {o:>8.4f} {o-b:>+8.4f}")
```

## 10. Limitations
Trained on a small curated dataset (~16,000 augmented pairs). May not generalize well
to domains far from the STS benchmark distribution (news, forums, headlines, image captions).
Symmetric augmentation assumes similarity is perfectly symmetric. This is generally true
for semantic similarity but may not hold for all retrieval scenarios.
Inherits biases from the base MiniLM architecture and SemEval training data, which is
predominantly English, web-sourced text.
Should not be used as the sole decision-making system in high-stakes applications
without human oversight.


## 11. License
This fine-tuned model is released under the Apache 2.0 License, consistent with the
license of the base model
sentence-transformers/all-MiniLM-L6-v2.
The training datasets (STS Benchmark, STS12) are released for research and evaluation
purposes by their respective authors. The raw dataset files are not redistributed here.


## 12. Credits and Citations
Base Model:
Nils Reimers and Iryna Gurevych.
"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks."
Proceedings of EMNLP 2019.
Model Card
STS Benchmark:
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, Lucia Specia.
"SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation."
Proceedings of SemEval-2017.
STS12:
Eneko Agirre, Daniel Cer, Mona Diab, Inigo Lopez-Gazpio, Lucia Specia.
"SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity."
Proceedings of *SEM 2012.
Evaluation Framework:
MTEB: Massive Text Embedding Benchmark


## 13. Contact
For questions, bug reports, or collaboration inquiries:
Email: blueprintai.help1@gmail.com
<p align="center">
<sub>Trained on consumer hardware. No datacenter required.</sub>
</p>