Update README.md

b7d6b68 verified 11 days ago

18.8 kB

	---
	language:
	- en
	license: apache-2.0
	library_name: sentence-transformers
	tags:
	- sentence-transformers
	- sentence-similarity
	- feature-extraction
	- sts
	- embeddings
	- symmetric-augmentation
	- fine-tuned
	base_model: sentence-transformers/all-MiniLM-L6-v2
	pipeline_tag: sentence-similarity
	datasets:
	- mteb/stsbenchmark-sts
	- mteb/sts12-sts
	metrics:
	- spearmanr
	---

	<h1 align="center">SymSTS-MiniLM-L6</h1>

	<p align="center">
	<strong>Symmetrically Augmented MiniLM for Semantic Textual Similarity</strong>
	</p>

	<p align="center">
	<img src="https://img.shields.io/badge/Parameters-22.7M-blue" alt="params"/>
	<img src="https://img.shields.io/badge/Embedding--Dim-384-orange" alt="dim"/>
	<img src="https://img.shields.io/badge/Max--Seq--Len-256-yellow" alt="seqlen"/>
	<img src="https://img.shields.io/badge/License-Apache--2.0-green" alt="license"/>
	<img src="https://img.shields.io/badge/STS--Avg--Spearman-0.8205-brightgreen" alt="score"/>
	</p>

	---

	## 1. What This Model Is

	SymSTS-MiniLM-L6 is a fine-tuned version of
	[sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2),
	specialized for Semantic Textual Similarity (STS).

	The key idea behind this model is symmetric data augmentation. During training, every sentence
	pair `(A, B, score)` is duplicated as `(B, A, score)`. This forces the model to learn that
	similarity is bidirectional: `similarity(A, B) = similarity(B, A)`. This simple technique
	produces consistent gains across all historical STS benchmarks.

	The model has 22.7 million parameters, runs fast on CPU, and fits comfortably on consumer GPUs.

	> This is NOT a general-purpose embedding model.
	> It is built for one job: measuring how similar two sentences are in meaning.
	> It is not optimized for document retrieval, classification, or multilingual tasks.

	---

	## 2. Model Architecture

	<table>
	<tr><th>Property</th><th>Value</th></tr>
	<tr><td>Base Architecture</td><td>MiniLM (6-layer Transformer encoder)</td></tr>
	<tr><td>Total Parameters</td><td>22.7 M</td></tr>
	<tr><td>Hidden Dimension</td><td>384</td></tr>
	<tr><td>Output Embedding Dimension</td><td>384</td></tr>
	<tr><td>Max Sequence Length</td><td>256 tokens</td></tr>
	<tr><td>Pooling Strategy</td><td>Mean Pooling</td></tr>
	<tr><td>Normalization</td><td>L2-normalized embeddings</td></tr>
	<tr><td>Similarity Function</td><td>Cosine Similarity</td></tr>
	</table>

	---

	## 3. Intended Use

	<table>
	<tr>
	<th>Good For</th>
	<th>Not Built For</th>
	</tr>
	<tr>
	<td>Semantic similarity scoring between two sentences</td>
	<td>Document retrieval or search ranking over large corpora</td>
	</tr>
	<tr>
	<td>Paraphrase and duplicate question detection</td>
	<td>Sentiment analysis or text classification</td>
	</tr>
	<tr>
	<td>Small-scale text clustering by meaning</td>
	<td>Zero-shot classification</td>
	</tr>
	<tr>
	<td>FAQ matching and chatbot intent matching</td>
	<td>Multilingual or cross-lingual tasks</td>
	</tr>
	<tr>
	<td>Sentence-level deduplication pipelines</td>
	<td>Long-document embedding beyond 256 tokens</td>
	</tr>
	</table>

	---

	## 4. Benchmark Results

	All scores are Spearman rank correlation measured locally using the
	[MTEB](https://github.com/embeddings-benchmark/mteb) library on official test splits.

	### 4.1 STS Results vs Base Model

	<table>
	<tr>
	<th>Task</th>
	<th>Base MiniLM-L6-v2</th>
	<th>SymSTS-MiniLM-L6 (Ours)</th>
	<th>Difference</th>
	</tr>
	<tr><td>STS12</td><td>0.7237</td><td><strong>0.7873</strong></td><td>+0.0636</td></tr>
	<tr><td>STS13</td><td>0.8060</td><td><strong>0.8290</strong></td><td>+0.0230</td></tr>
	<tr><td>STS14</td><td>0.7559</td><td><strong>0.8186</strong></td><td>+0.0627</td></tr>
	<tr><td>STS15</td><td>0.8539</td><td><strong>0.8758</strong></td><td>+0.0219</td></tr>
	<tr><td>STS16</td><td>0.7899</td><td><strong>0.8152</strong></td><td>+0.0253</td></tr>
	<tr><td>STSBenchmark</td><td>0.8203</td><td><strong>0.8407</strong></td><td>+0.0204</td></tr>
	<tr><td>SICK-R</td><td>0.7758</td><td><strong>0.7772</strong></td><td>+0.0013</td></tr>
	<tr>
	<td><strong>Average</strong></td>
	<td><strong>0.7894</strong></td>
	<td><strong>0.8205</strong></td>
	<td><strong>+0.0312</strong></td>
	</tr>
	</table>

	### 4.2 Honest Comparison Against Other Models

	Scores for external models are approximate, taken from the MTEB leaderboard and published
	model cards. Our scores are measured locally. We include models that outperform ours
	for full transparency.

	<table>
	<tr>
	<th>Model</th>
	<th>Params</th>
	<th>STS-B Spearman</th>
	<th>STS Avg (approx)</th>
	<th>vs SymSTS</th>
	</tr>
	<tr>
	<td><strong>SymSTS-MiniLM-L6 (Ours)</strong></td>
	<td>22.7M</td>
	<td><strong>0.8407</strong></td>
	<td><strong>0.8205</strong></td>
	<td>--</td>
	</tr>
	<tr>
	<td>all-MiniLM-L6-v2</td>
	<td>22.7M</td>
	<td>0.8203</td>
	<td>0.7894</td>
	<td>SymSTS wins</td>
	</tr>
	<tr>
	<td>all-MiniLM-L12-v2</td>
	<td>33M</td>
	<td>~0.835</td>
	<td>~0.805</td>
	<td>SymSTS wins</td>
	</tr>
	<tr>
	<td>all-mpnet-base-v2</td>
	<td>109M</td>
	<td>~0.835</td>
	<td>~0.810</td>
	<td>SymSTS wins</td>
	</tr>
	<tr>
	<td>bge-small-en-v1.5</td>
	<td>33M</td>
	<td>~0.815</td>
	<td>~0.800</td>
	<td>SymSTS wins</td>
	</tr>
	<tr>
	<td>e5-small-v2</td>
	<td>33M</td>
	<td>~0.820</td>
	<td>~0.805</td>
	<td>SymSTS wins</td>
	</tr>
	<tr>
	<td>gte-small</td>
	<td>33M</td>
	<td>~0.840</td>
	<td>~0.815</td>
	<td>Comparable</td>
	</tr>
	<tr>
	<td>bge-base-en-v1.5</td>
	<td>110M</td>
	<td>~0.855</td>
	<td>~0.835</td>
	<td>They win</td>
	</tr>
	<tr>
	<td>gte-base</td>
	<td>110M</td>
	<td>~0.855</td>
	<td>~0.840</td>
	<td>They win</td>
	</tr>
	<tr>
	<td>nomic-embed-text-v1.5</td>
	<td>137M</td>
	<td>~0.865</td>
	<td>~0.845</td>
	<td>They win</td>
	</tr>
	<tr>
	<td>jina-embeddings-v3</td>
	<td>570M</td>
	<td>~0.870</td>
	<td>~0.855</td>
	<td>They win</td>
	</tr>
	<tr>
	<td>e5-mistral-7b-instruct</td>
	<td>7B</td>
	<td>~0.880</td>
	<td>~0.860</td>
	<td>They win</td>
	</tr>
	</table>

	Summary: SymSTS-MiniLM-L6 outperforms all models at or below 33M parameters and several
	models up to 109M parameters on STS tasks. It is outperformed by larger base-architecture
	models (110M+) and LLM-based embeddings, which is expected given the parameter gap.

	---

	## 5. Training Details

	### 5.1 Loss Function

	`CosineSimilarityLoss` — minimizes the mean squared error between the predicted cosine
	similarity and the human-annotated similarity score.

	### 5.2 Symmetric Data Augmentation

	For every training pair:
	Original: (Sentence_A, Sentence_B, score)
	Augmented: (Sentence_B, Sentence_A, score)

	Both are included, doubling the effective training set from ~8k to ~16k pairs. Note: because cosine similarity is symmetric by construction (sim(A,B) = sim(B,A) regardless of training), this augmentation's main effect is increasing the volume of training pairs rather than teaching the model a new bidirectional property. The performance gains reported in Section 4 are real and measured directly; the ablation isolating "more data" from "swap specifically" is planned as future work.
	### 5.3 Hyperparameters

	<table>
	<tr><th>Hyperparameter</th><th>Value</th></tr>
	<tr><td>Learning Rate</td><td>1e-5</td></tr>
	<tr><td>Batch Size</td><td>16</td></tr>
	<tr><td>Epochs</td><td>2</td></tr>
	<tr><td>Warmup</td><td>10% of total steps</td></tr>
	<tr><td>Weight Decay</td><td>0.01</td></tr>
	<tr><td>Optimizer</td><td>AdamW</td></tr>
	<tr><td>Precision</td><td>FP16 (mixed precision)</td></tr>
	<tr><td>Random Seed</td><td>42</td></tr>
	<tr><td>Hardware</td><td>NVIDIA GeForce GTX 1660 SUPER (6 GB VRAM)</td></tr>
	</table>

	---

	## 6. Training Data

	The model was fine-tuned on the train splits of the following datasets.
	STS13 through STS16 and SICK-R were not used during training. They are held-out
	evaluation benchmarks only.

	<table>
	<tr>
	<th>Dataset</th>
	<th>Source</th>
	<th>Original Pairs</th>
	<th>After Augmentation</th>
	<th>License</th>
	</tr>
	<tr>
	<td>STS Benchmark (train)</td>
	<td>SemEval-2017 Task 1</td>
	<td>5,749</td>
	<td>11,498</td>
	<td>Research use</td>
	</tr>
	<tr>
	<td>STS12 (train)</td>
	<td>SemEval-2012 Task 6</td>
	<td>2,234</td>
	<td>4,468</td>
	<td>Research use</td>
	</tr>
	<tr>
	<td><strong>Total</strong></td>
	<td></td>
	<td><strong>7,983</strong></td>
	<td><strong>15,966</strong></td>
	<td></td>
	</tr>
	</table>

	The raw dataset files are not redistributed in this repository.
	Only the fine-tuned model weights are provided.

	---

	## 7. Usage

	```python
	from sentence_transformers import SentenceTransformer, util

	model = SentenceTransformer("blueprint-ai/SymSTS-MiniLM")

	sentences = [
	"The cat sits on the mat.",
	"A feline is resting on the rug.",
	"The stock market crashed today."
	]

	embeddings = model.encode(sentences, normalize_embeddings=True)

	sim_01 = util.cos_sim(embeddings[0], embeddings[1]).item()
	sim_02 = util.cos_sim(embeddings[0], embeddings[2]).item()

	print(f"Cat vs Feline: {sim_01:.4f}")
	print(f"Cat vs Stocks: {sim_02:.4f}")
	#8. Full Training Code
	This is the exact script used to train SymSTS-MiniLM-L6.
	import warnings
	warnings.filterwarnings("ignore")

	import os
	import torch
	from datasets import load_dataset, Dataset

	from sentence_transformers import (
	SentenceTransformer,
	SentenceTransformerTrainer,
	SentenceTransformerTrainingArguments,
	InputExample,
	)
	from sentence_transformers.losses import CosineSimilarityLoss
	from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator


	# =============================================================
	# Configuration
	# =============================================================
	BASE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
	OUTPUT_DIR = "./output/symsts-minilm-l6"
	RUNS_DIR = "./output/runs-symsts-minilm-l6"
	MAX_SEQ_LEN = 256
	BATCH_SIZE = 16
	LEARNING_RATE = 1e-5
	EPOCHS = 2

	device = "cuda" if torch.cuda.is_available() else "cpu"
	print(f"Device: {device}")


	# =============================================================
	# Helpers
	# =============================================================
	def clean_text(text):
	if text is None:
	return ""
	return " ".join(str(text).strip().split())


	def get_main_score(result):
	if isinstance(result, dict):
	for key, value in result.items():
	if "spearman_cosine" in key:
	try: return float(value)
	except Exception: pass
	for key, value in result.items():
	if "spearman" in key:
	try: return float(value)
	except Exception: pass
	for value in result.values():
	try: return float(value)
	except Exception: pass
	try: return float(result)
	except Exception: return 0.0


	# =============================================================
	# 1. Load Base Model
	# =============================================================
	print("Loading base model...")
	student = SentenceTransformer(BASE_MODEL, device=device)
	student.max_seq_length = MAX_SEQ_LEN


	# =============================================================
	# 2. Load STS Training Data with Symmetric Augmentation
	# =============================================================
	print("Loading STS training data...")

	sources = [
	("mteb/stsbenchmark-sts", "train"),
	("mteb/sts12-sts", "train"),
	]

	sentence1_list = []
	sentence2_list = []
	label_list = []
	seen_pairs = set()


	def add_pair(s1, s2, score):
	s1 = clean_text(s1)
	s2 = clean_text(s2)
	if not s1 or not s2:
	return

	key = tuple(sorted((s1.lower(), s2.lower())))
	if key in seen_pairs:
	return
	seen_pairs.add(key)

	# Forward
	sentence1_list.append(s1)
	sentence2_list.append(s2)
	label_list.append(score)

	# Reverse (Symmetric Augmentation)
	if s1.lower() != s2.lower():
	sentence1_list.append(s2)
	sentence2_list.append(s1)
	label_list.append(score)


	for dataset_name, split in sources:
	try:
	ds = load_dataset(dataset_name, split=split)
	count = 0
	for row in ds:
	s1 = row.get("sentence1")
	s2 = row.get("sentence2")
	score = row.get("score", 0.0)
	try:
	score = float(score)
	except Exception:
	continue
	if score > 1.0:
	score = score / 5.0
	score = max(0.0, min(1.0, score))
	add_pair(s1, s2, score)
	count += 1
	print(f" Loaded {count} original pairs from {dataset_name}")
	except Exception as e:
	print(f" Skipping {dataset_name}: {e}")

	print(f"Unique pairs: {len(seen_pairs)}")
	print(f"Total augmented rows: {len(sentence1_list)}")

	train_dataset = Dataset.from_dict({
	"sentence1": sentence1_list,
	"sentence2": sentence2_list,
	"label": label_list,
	})


	# =============================================================
	# 3. STS-B Test Evaluator
	# =============================================================
	print("Loading STS-B test evaluator...")

	sts_test = load_dataset("mteb/stsbenchmark-sts", split="test")
	eval_examples = []
	for row in sts_test:
	s1 = clean_text(row.get("sentence1"))
	s2 = clean_text(row.get("sentence2"))
	score = float(row.get("score", 0.0)) / 5.0
	eval_examples.append(InputExample(texts=[s1, s2], label=score))

	evaluator = EmbeddingSimilarityEvaluator.from_input_examples(
	eval_examples,
	name="sts-b-test",
	)

	base_score = get_main_score(evaluator(student))
	print(f"Base model STS-B score: {base_score:.4f}")


	# =============================================================
	# 4. Train
	# =============================================================
	print("Training SymSTS-MiniLM-L6...")

	train_loss = CosineSimilarityLoss(model=student)

	training_args = SentenceTransformerTrainingArguments(
	output_dir = RUNS_DIR,
	num_train_epochs = EPOCHS,
	per_device_train_batch_size = BATCH_SIZE,
	per_device_eval_batch_size = BATCH_SIZE,
	learning_rate = LEARNING_RATE,
	warmup_steps = 0.1,
	weight_decay = 0.01,
	fp16 = torch.cuda.is_available(),
	bf16 = False,
	logging_steps = 50,
	save_strategy = "no",
	eval_strategy = "epoch",
	dataloader_num_workers = 0,
	report_to = "none",
	remove_unused_columns = False,
	seed = 42,
	)

	trainer = SentenceTransformerTrainer(
	model = student,
	args = training_args,
	train_dataset = train_dataset,
	loss = train_loss,
	evaluator = evaluator,
	)

	trainer.train()

	os.makedirs(OUTPUT_DIR, exist_ok=True)
	try:
	trainer.save_model(OUTPUT_DIR)
	except Exception:
	student.save_pretrained(OUTPUT_DIR)

	print(f"Model saved to: {OUTPUT_DIR}")


	# =============================================================
	# 5. Final Evaluation
	# =============================================================
	final_score = get_main_score(evaluator(student))

	print("=" * 50)
	print(f"Base model STS-B: {base_score:.4f}")
	print(f"SymSTS STS-B: {final_score:.4f}")
	print(f"Improvement: {final_score - base_score:+.4f}")
	print("=" * 50)

	#9. Evaluation Code
	This is the script used to produce the benchmark tables above.
	import warnings
	warnings.filterwarnings("ignore")

	import os, json, glob, mteb
	from sentence_transformers import SentenceTransformer

	os.environ["TOKENIZERS_PARALLELISM"] = "false"

	TASKS = [
	"STS12", "STS13", "STS14", "STS15",
	"STS16", "STSBenchmark", "SICK-R",
	]

	def get_scores(model_path, out_dir):
	model = SentenceTransformer(model_path)
	tasks = mteb.get_tasks(tasks=TASKS)
	evaluation = mteb.MTEB(tasks=tasks)
	evaluation.run(model, output_folder=out_dir, verbosity=0)

	scores = {}
	for task_name in TASKS:
	files = glob.glob(f"{out_dir}/*/{task_name}*.json", recursive=True)
	if files:
	with open(files[0]) as f:
	d = json.load(f)
	if "scores" in d and "test" in d["scores"]:
	scores[task_name] = d["scores"]["test"][0].get("main_score", 0.0)
	return scores

	base = get_scores("sentence-transformers/all-MiniLM-L6-v2", "./mteb_base")
	ours = get_scores("./output/symsts-minilm-l6", "./mteb_ours")

	print(f"{'Task':<16} {'Base':>8} {'Ours':>8} {'Diff':>8}")
	print("-" * 44)
	for t in TASKS:
	b = base.get(t, 0)
	o = ours.get(t, 0)
	print(f"{t:<16} {b:>8.4f} {o:>8.4f} {o-b:>+8.4f}")
	```

	## 10. Limitations
	Trained on a small curated dataset (~16,000 augmented pairs). May not generalize well
	to domains far from the STS benchmark distribution (news, forums, headlines, image captions).
	Symmetric augmentation assumes similarity is perfectly symmetric. This is generally true
	for semantic similarity but may not hold for all retrieval scenarios.
	Inherits biases from the base MiniLM architecture and SemEval training data, which is
	predominantly English, web-sourced text.
	Should not be used as the sole decision-making system in high-stakes applications
	without human oversight.


	## 11. License
	This fine-tuned model is released under the Apache 2.0 License, consistent with the
	license of the base model
	sentence-transformers/all-MiniLM-L6-v2.
	The training datasets (STS Benchmark, STS12) are released for research and evaluation
	purposes by their respective authors. The raw dataset files are not redistributed here.


	## 12. Credits and Citations
	Base Model:
	Nils Reimers and Iryna Gurevych.
	"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks."
	Proceedings of EMNLP 2019.
	Model Card
	STS Benchmark:
	Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, Lucia Specia.
	"SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation."
	Proceedings of SemEval-2017.
	STS12:
	Eneko Agirre, Daniel Cer, Mona Diab, Inigo Lopez-Gazpio, Lucia Specia.
	"SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity."
	Proceedings of *SEM 2012.
	Evaluation Framework:
	MTEB: Massive Text Embedding Benchmark


	## 13. Contact
	For questions, bug reports, or collaboration inquiries:
	Email: blueprintai.help1@gmail.com
	<p align="center">
	<sub>Trained on consumer hardware. No datacenter required.</sub>
	</p>