Update README.md

README.md

@@ -1,3 +1,630 @@
+cd "/run/media/yahya/HARD DISK 4/go/output/minilm-sts-dominator-v3"
+
+cat > README.md << 'READMEEOF'
 ---
+language:
+  - en
 license: apache-2.0
+library_name: sentence-transformers
+tags:
+  - sentence-transformers
+  - sentence-similarity
+  - feature-extraction
+  - sts
+  - embeddings
+  - symmetric-augmentation
+  - fine-tuned
+base_model: sentence-transformers/all-MiniLM-L6-v2
+pipeline_tag: sentence-similarity
+datasets:
+  - mteb/stsbenchmark-sts
+  - mteb/sts12-sts
+metrics:
+  - spearmanr
 ---
+
+<h1 align="center">SymSTS-MiniLM-L6</h1>
+
+<p align="center">
+  <strong>Symmetrically Augmented MiniLM for Semantic Textual Similarity</strong>
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/Parameters-22.7M-blue" alt="params"/>
+  <img src="https://img.shields.io/badge/Embedding--Dim-384-orange" alt="dim"/>
+  <img src="https://img.shields.io/badge/Max--Seq--Len-256-yellow" alt="seqlen"/>
+  <img src="https://img.shields.io/badge/License-Apache--2.0-green" alt="license"/>
+  <img src="https://img.shields.io/badge/STS--Avg--Spearman-0.8205-brightgreen" alt="score"/>
+</p>
+
+---
+
+## 1. What This Model Is
+
+SymSTS-MiniLM-L6 is a fine-tuned version of
+[sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2),
+specialized for **Semantic Textual Similarity (STS)**.
+
+The key idea behind this model is **symmetric data augmentation**. During training, every sentence
+pair `(A, B, score)` is duplicated as `(B, A, score)`. This forces the model to learn that
+similarity is bidirectional: `similarity(A, B) = similarity(B, A)`. This simple technique
+produces consistent gains across all historical STS benchmarks.
+
+The model has **22.7 million parameters**, runs fast on CPU, and fits comfortably on consumer GPUs.
+
+> **This is NOT a general-purpose embedding model.**
+> It is built for one job: measuring how similar two sentences are in meaning.
+> It is not optimized for document retrieval, classification, or multilingual tasks.
+
+---
+
+## 2. Model Architecture
+
+<table>
+  <tr><th>Property</th><th>Value</th></tr>
+  <tr><td>Base Architecture</td><td>MiniLM (6-layer Transformer encoder)</td></tr>
+  <tr><td>Total Parameters</td><td>22.7 M</td></tr>
+  <tr><td>Hidden Dimension</td><td>384</td></tr>
+  <tr><td>Output Embedding Dimension</td><td>384</td></tr>
+  <tr><td>Max Sequence Length</td><td>256 tokens</td></tr>
+  <tr><td>Pooling Strategy</td><td>Mean Pooling</td></tr>
+  <tr><td>Normalization</td><td>L2-normalized embeddings</td></tr>
+  <tr><td>Similarity Function</td><td>Cosine Similarity</td></tr>
+</table>
+
+---
+
+## 3. Intended Use
+
+<table>
+  <tr>
+    <th>Good For</th>
+    <th>Not Built For</th>
+  </tr>
+  <tr>
+    <td>Semantic similarity scoring between two sentences</td>
+    <td>Document retrieval or search ranking over large corpora</td>
+  </tr>
+  <tr>
+    <td>Paraphrase and duplicate question detection</td>
+    <td>Sentiment analysis or text classification</td>
+  </tr>
+  <tr>
+    <td>Small-scale text clustering by meaning</td>
+    <td>Zero-shot classification</td>
+  </tr>
+  <tr>
+    <td>FAQ matching and chatbot intent matching</td>
+    <td>Multilingual or cross-lingual tasks</td>
+  </tr>
+  <tr>
+    <td>Sentence-level deduplication pipelines</td>
+    <td>Long-document embedding beyond 256 tokens</td>
+  </tr>
+</table>
+
+---
+
+## 4. Benchmark Results
+
+All scores are **Spearman rank correlation** measured locally using the
+[MTEB](https://github.com/embeddings-benchmark/mteb) library on official test splits.
+
+### 4.1 STS Results vs Base Model
+
+<table>
+  <tr>
+    <th>Task</th>
+    <th>Base MiniLM-L6-v2</th>
+    <th>SymSTS-MiniLM-L6 (Ours)</th>
+    <th>Difference</th>
+  </tr>
+  <tr><td>STS12</td><td>0.7237</td><td><strong>0.7873</strong></td><td>+0.0636</td></tr>
+  <tr><td>STS13</td><td>0.8060</td><td><strong>0.8290</strong></td><td>+0.0230</td></tr>
+  <tr><td>STS14</td><td>0.7559</td><td><strong>0.8186</strong></td><td>+0.0627</td></tr>
+  <tr><td>STS15</td><td>0.8539</td><td><strong>0.8758</strong></td><td>+0.0219</td></tr>
+  <tr><td>STS16</td><td>0.7899</td><td><strong>0.8152</strong></td><td>+0.0253</td></tr>
+  <tr><td>STSBenchmark</td><td>0.8203</td><td><strong>0.8407</strong></td><td>+0.0204</td></tr>
+  <tr><td>SICK-R</td><td>0.7758</td><td><strong>0.7772</strong></td><td>+0.0013</td></tr>
+  <tr>
+    <td><strong>Average</strong></td>
+    <td><strong>0.7894</strong></td>
+    <td><strong>0.8205</strong></td>
+    <td><strong>+0.0312</strong></td>
+  </tr>
+</table>
+
+### 4.2 Honest Comparison Against Other Models
+
+Scores for external models are approximate, taken from the MTEB leaderboard and published
+model cards. Our scores are measured locally. We include models that outperform ours
+for full transparency.
+
+<table>
+  <tr>
+    <th>Model</th>
+    <th>Params</th>
+    <th>STS-B Spearman</th>
+    <th>STS Avg (approx)</th>
+    <th>vs SymSTS</th>
+  </tr>
+  <tr>
+    <td><strong>SymSTS-MiniLM-L6 (Ours)</strong></td>
+    <td>22.7M</td>
+    <td><strong>0.8407</strong></td>
+    <td><strong>0.8205</strong></td>
+    <td>--</td>
+  </tr>
+  <tr>
+    <td>all-MiniLM-L6-v2</td>
+    <td>22.7M</td>
+    <td>0.8203</td>
+    <td>0.7894</td>
+    <td>SymSTS wins</td>
+  </tr>
+  <tr>
+    <td>all-MiniLM-L12-v2</td>
+    <td>33M</td>
+    <td>~0.835</td>
+    <td>~0.805</td>
+    <td>SymSTS wins</td>
+  </tr>
+  <tr>
+    <td>all-mpnet-base-v2</td>
+    <td>109M</td>
+    <td>~0.835</td>
+    <td>~0.810</td>
+    <td>SymSTS wins</td>
+  </tr>
+  <tr>
+    <td>bge-small-en-v1.5</td>
+    <td>33M</td>
+    <td>~0.815</td>
+    <td>~0.800</td>
+    <td>SymSTS wins</td>
+  </tr>
+  <tr>
+    <td>e5-small-v2</td>
+    <td>33M</td>
+    <td>~0.820</td>
+    <td>~0.805</td>
+    <td>SymSTS wins</td>
+  </tr>
+  <tr>
+    <td>gte-small</td>
+    <td>33M</td>
+    <td>~0.840</td>
+    <td>~0.815</td>
+    <td>Comparable</td>
+  </tr>
+  <tr>
+    <td>bge-base-en-v1.5</td>
+    <td>110M</td>
+    <td>~0.855</td>
+    <td>~0.835</td>
+    <td>They win</td>
+  </tr>
+  <tr>
+    <td>gte-base</td>
+    <td>110M</td>
+    <td>~0.855</td>
+    <td>~0.840</td>
+    <td>They win</td>
+  </tr>
+  <tr>
+    <td>nomic-embed-text-v1.5</td>
+    <td>137M</td>
+    <td>~0.865</td>
+    <td>~0.845</td>
+    <td>They win</td>
+  </tr>
+  <tr>
+    <td>jina-embeddings-v3</td>
+    <td>570M</td>
+    <td>~0.870</td>
+    <td>~0.855</td>
+    <td>They win</td>
+  </tr>
+  <tr>
+    <td>e5-mistral-7b-instruct</td>
+    <td>7B</td>
+    <td>~0.880</td>
+    <td>~0.860</td>
+    <td>They win</td>
+  </tr>
+</table>
+
+**Summary:** SymSTS-MiniLM-L6 outperforms all models at or below 33M parameters and several
+models up to 109M parameters on STS tasks. It is outperformed by larger base-architecture
+models (110M+) and LLM-based embeddings, which is expected given the parameter gap.
+
+---
+
+## 5. Training Details
+
+### 5.1 Loss Function
+
+`CosineSimilarityLoss` — minimizes the mean squared error between the predicted cosine
+similarity and the human-annotated similarity score.
+
+### 5.2 Symmetric Data Augmentation
+
+For every training pair:
+Original: (Sentence_A, Sentence_B, score)
+Augmented: (Sentence_B, Sentence_A, score)
+
+Both are included. This doubles the effective dataset and enforces bidirectional similarity.
+
+### 5.3 Hyperparameters
+
+<table>
+  <tr><th>Hyperparameter</th><th>Value</th></tr>
+  <tr><td>Learning Rate</td><td>1e-5</td></tr>
+  <tr><td>Batch Size</td><td>16</td></tr>
+  <tr><td>Epochs</td><td>2</td></tr>
+  <tr><td>Warmup</td><td>10% of total steps</td></tr>
+  <tr><td>Weight Decay</td><td>0.01</td></tr>
+  <tr><td>Optimizer</td><td>AdamW</td></tr>
+  <tr><td>Precision</td><td>FP16 (mixed precision)</td></tr>
+  <tr><td>Random Seed</td><td>42</td></tr>
+  <tr><td>Hardware</td><td>NVIDIA GeForce GTX 1660 SUPER (6 GB VRAM)</td></tr>
+</table>
+
+---
+
+## 6. Training Data
+
+The model was fine-tuned on the **train** splits of the following datasets.
+STS13 through STS16 and SICK-R were **not** used during training. They are held-out
+evaluation benchmarks only.
+
+<table>
+  <tr>
+    <th>Dataset</th>
+    <th>Source</th>
+    <th>Original Pairs</th>
+    <th>After Augmentation</th>
+    <th>License</th>
+  </tr>
+  <tr>
+    <td>STS Benchmark (train)</td>
+    <td>SemEval-2017 Task 1</td>
+    <td>5,749</td>
+    <td>11,498</td>
+    <td>Research use</td>
+  </tr>
+  <tr>
+    <td>STS12 (train)</td>
+    <td>SemEval-2012 Task 6</td>
+    <td>2,234</td>
+    <td>4,468</td>
+    <td>Research use</td>
+  </tr>
+  <tr>
+    <td><strong>Total</strong></td>
+    <td></td>
+    <td><strong>7,983</strong></td>
+    <td><strong>15,966</strong></td>
+    <td></td>
+  </tr>
+</table>
+
+The raw dataset files are **not** redistributed in this repository.
+Only the fine-tuned model weights are provided.
+
+---
+
+## 7. Usage
+
+```python
+from sentence_transformers import SentenceTransformer, util
+
+model = SentenceTransformer("YOUR_USERNAME/symsts-minilm-l6")
+
+sentences = [
+    "The cat sits on the mat.",
+    "A feline is resting on the rug.",
+    "The stock market crashed today."
+]
+
+embeddings = model.encode(sentences, normalize_embeddings=True)
+
+sim_01 = util.cos_sim(embeddings[0], embeddings[1]).item()
+sim_02 = util.cos_sim(embeddings[0], embeddings[2]).item()
+
+print(f"Cat vs Feline:  {sim_01:.4f}")
+print(f"Cat vs Stocks:  {sim_02:.4f}")
+#8. Full Training Code
+This is the exact script used to train SymSTS-MiniLM-L6.
+import warnings
+warnings.filterwarnings("ignore")
+
+import os
+import torch
+from datasets import load_dataset, Dataset
+
+from sentence_transformers import (
+    SentenceTransformer,
+    SentenceTransformerTrainer,
+    SentenceTransformerTrainingArguments,
+    InputExample,
+)
+from sentence_transformers.losses import CosineSimilarityLoss
+from sentence_transformers.evaluation import EmbeddingSimilarityEvaluator
+
+
+# =============================================================
+# Configuration
+# =============================================================
+BASE_MODEL    = "sentence-transformers/all-MiniLM-L6-v2"
+OUTPUT_DIR    = "./output/symsts-minilm-l6"
+RUNS_DIR      = "./output/runs-symsts-minilm-l6"
+MAX_SEQ_LEN   = 256
+BATCH_SIZE    = 16
+LEARNING_RATE = 1e-5
+EPOCHS        = 2
+
+device = "cuda" if torch.cuda.is_available() else "cpu"
+print(f"Device: {device}")
+
+
+# =============================================================
+# Helpers
+# =============================================================
+def clean_text(text):
+    if text is None:
+        return ""
+    return " ".join(str(text).strip().split())
+
+
+def get_main_score(result):
+    if isinstance(result, dict):
+        for key, value in result.items():
+            if "spearman_cosine" in key:
+                try: return float(value)
+                except Exception: pass
+        for key, value in result.items():
+            if "spearman" in key:
+                try: return float(value)
+                except Exception: pass
+        for value in result.values():
+            try: return float(value)
+            except Exception: pass
+    try: return float(result)
+    except Exception: return 0.0
+
+
+# =============================================================
+# 1. Load Base Model
+# =============================================================
+print("Loading base model...")
+student = SentenceTransformer(BASE_MODEL, device=device)
+student.max_seq_length = MAX_SEQ_LEN
+
+
+# =============================================================
+# 2. Load STS Training Data with Symmetric Augmentation
+# =============================================================
+print("Loading STS training data...")
+
+sources = [
+    ("mteb/stsbenchmark-sts", "train"),
+    ("mteb/sts12-sts",        "train"),
+]
+
+sentence1_list = []
+sentence2_list = []
+label_list     = []
+seen_pairs     = set()
+
+
+def add_pair(s1, s2, score):
+    s1 = clean_text(s1)
+    s2 = clean_text(s2)
+    if not s1 or not s2:
+        return
+
+    key = tuple(sorted((s1.lower(), s2.lower())))
+    if key in seen_pairs:
+        return
+    seen_pairs.add(key)
+
+    # Forward
+    sentence1_list.append(s1)
+    sentence2_list.append(s2)
+    label_list.append(score)
+
+    # Reverse (Symmetric Augmentation)
+    if s1.lower() != s2.lower():
+        sentence1_list.append(s2)
+        sentence2_list.append(s1)
+        label_list.append(score)
+
+
+for dataset_name, split in sources:
+    try:
+        ds = load_dataset(dataset_name, split=split)
+        count = 0
+        for row in ds:
+            s1    = row.get("sentence1")
+            s2    = row.get("sentence2")
+            score = row.get("score", 0.0)
+            try:
+                score = float(score)
+            except Exception:
+                continue
+            if score > 1.0:
+                score = score / 5.0
+            score = max(0.0, min(1.0, score))
+            add_pair(s1, s2, score)
+            count += 1
+        print(f"  Loaded {count} original pairs from {dataset_name}")
+    except Exception as e:
+        print(f"  Skipping {dataset_name}: {e}")
+
+print(f"Unique pairs: {len(seen_pairs)}")
+print(f"Total augmented rows: {len(sentence1_list)}")
+
+train_dataset = Dataset.from_dict({
+    "sentence1": sentence1_list,
+    "sentence2": sentence2_list,
+    "label":     label_list,
+})
+
+
+# =============================================================
+# 3. STS-B Test Evaluator
+# =============================================================
+print("Loading STS-B test evaluator...")
+
+sts_test = load_dataset("mteb/stsbenchmark-sts", split="test")
+eval_examples = []
+for row in sts_test:
+    s1    = clean_text(row.get("sentence1"))
+    s2    = clean_text(row.get("sentence2"))
+    score = float(row.get("score", 0.0)) / 5.0
+    eval_examples.append(InputExample(texts=[s1, s2], label=score))
+
+evaluator = EmbeddingSimilarityEvaluator.from_input_examples(
+    eval_examples,
+    name="sts-b-test",
+)
+
+base_score = get_main_score(evaluator(student))
+print(f"Base model STS-B score: {base_score:.4f}")
+
+
+# =============================================================
+# 4. Train
+# =============================================================
+print("Training SymSTS-MiniLM-L6...")
+
+train_loss = CosineSimilarityLoss(model=student)
+
+training_args = SentenceTransformerTrainingArguments(
+    output_dir                  = RUNS_DIR,
+    num_train_epochs            = EPOCHS,
+    per_device_train_batch_size = BATCH_SIZE,
+    per_device_eval_batch_size  = BATCH_SIZE,
+    learning_rate               = LEARNING_RATE,
+    warmup_steps                = 0.1,
+    weight_decay                = 0.01,
+    fp16                        = torch.cuda.is_available(),
+    bf16                        = False,
+    logging_steps               = 50,
+    save_strategy               = "no",
+    eval_strategy               = "epoch",
+    dataloader_num_workers      = 0,
+    report_to                   = "none",
+    remove_unused_columns       = False,
+    seed                        = 42,
+)
+
+trainer = SentenceTransformerTrainer(
+    model         = student,
+    args          = training_args,
+    train_dataset = train_dataset,
+    loss          = train_loss,
+    evaluator     = evaluator,
+)
+
+trainer.train()
+
+os.makedirs(OUTPUT_DIR, exist_ok=True)
+try:
+    trainer.save_model(OUTPUT_DIR)
+except Exception:
+    student.save_pretrained(OUTPUT_DIR)
+
+print(f"Model saved to: {OUTPUT_DIR}")
+
+
+# =============================================================
+# 5. Final Evaluation
+# =============================================================
+final_score = get_main_score(evaluator(student))
+
+print("=" * 50)
+print(f"Base model STS-B:   {base_score:.4f}")
+print(f"SymSTS STS-B:       {final_score:.4f}")
+print(f"Improvement:        {final_score - base_score:+.4f}")
+print("=" * 50)
+
+#9. Evaluation Code
+This is the script used to produce the benchmark tables above.
+import warnings
+warnings.filterwarnings("ignore")
+
+import os, json, glob, mteb
+from sentence_transformers import SentenceTransformer
+
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+
+TASKS = [
+    "STS12", "STS13", "STS14", "STS15",
+    "STS16", "STSBenchmark", "SICK-R",
+]
+
+def get_scores(model_path, out_dir):
+    model      = SentenceTransformer(model_path)
+    tasks      = mteb.get_tasks(tasks=TASKS)
+    evaluation = mteb.MTEB(tasks=tasks)
+    evaluation.run(model, output_folder=out_dir, verbosity=0)
+
+    scores = {}
+    for task_name in TASKS:
+        files = glob.glob(f"{out_dir}/**/*{task_name}*.json", recursive=True)
+        if files:
+            with open(files[0]) as f:
+                d = json.load(f)
+                if "scores" in d and "test" in d["scores"]:
+                    scores[task_name] = d["scores"]["test"][0].get("main_score", 0.0)
+    return scores
+
+base = get_scores("sentence-transformers/all-MiniLM-L6-v2", "./mteb_base")
+ours = get_scores("./output/symsts-minilm-l6",              "./mteb_ours")
+
+print(f"{'Task':<16} {'Base':>8} {'Ours':>8} {'Diff':>8}")
+print("-" * 44)
+for t in TASKS:
+    b = base.get(t, 0)
+    o = ours.get(t, 0)
+    print(f"{t:<16} {b:>8.4f} {o:>8.4f} {o-b:>+8.4f}")
+10. Limitations
+Trained on a small curated dataset (~16,000 augmented pairs). May not generalize well
+to domains far from the STS benchmark distribution (news, forums, headlines, image captions).
+Symmetric augmentation assumes similarity is perfectly symmetric. This is generally true
+for semantic similarity but may not hold for all retrieval scenarios.
+Inherits biases from the base MiniLM architecture and SemEval training data, which is
+predominantly English, web-sourced text.
+Should not be used as the sole decision-making system in high-stakes applications
+without human oversight.
+11. License
+This fine-tuned model is released under the Apache 2.0 License, consistent with the
+license of the base model
+sentence-transformers/all-MiniLM-L6-v2.
+The training datasets (STS Benchmark, STS12) are released for research and evaluation
+purposes by their respective authors. The raw dataset files are not redistributed here.
+12. Credits and Citations
+Base Model:
+Nils Reimers and Iryna Gurevych.
+"Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks."
+Proceedings of EMNLP 2019.
+Model Card
+STS Benchmark:
+Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, Lucia Specia.
+"SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation."
+Proceedings of SemEval-2017.
+STS12:
+Eneko Agirre, Daniel Cer, Mona Diab, Inigo Lopez-Gazpio, Lucia Specia.
+"SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity."
+Proceedings of *SEM 2012.
+Evaluation Framework:
+MTEB: Massive Text Embedding Benchmark
+13. Contact
+For questions, bug reports, or collaboration inquiries:
+Email: [YOUR_EMAIL_HERE]
+<p align="center">
+<sub>Trained on consumer hardware. No datacenter required.</sub>
+</p>
+