pyrawn's picture
Add new SentenceTransformer model
49a48bc verified
|
Raw History Blame Contribute Delete
6.6 kB
---
language: en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: google-bert/bert-base-uncased
datasets:
- stanfordnlp/snli
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- simcse
- contrastive-learning
model-index:
- name: simcse-bert-base-snli-unsup
results:
- task:
type: semantic-similarity
name: Semantic Textual Similarity
dataset:
type: sentence-transformers/stsb
name: STS-B (test)
split: test
metrics:
- type: spearman_cosine
value: 68.33
name: Spearman (cosine) x100
---
# simcse-bert-base-snli-unsup
Unsupervised SimCSE ([Gao, Yao & Chen, EMNLP 2021](https://arxiv.org/abs/2104.08821), Eq. 1) on top of `bert-base-uncased`, trained on **sentences from a 100k-record SNLI subset** as a course assignment (U2T02, Trends in Data Science). It maps a sentence to a 768-dimensional, L2-normalized vector; compare vectors with cosine similarity.
**Team:** Julio Cesar De Aquino Castellanos, Gustavo Alexander Fuentes Marin, Valeria Nicol Hernandez Leon, Ricardo Daniel Horta Sanchez, Jose Angel Pech Xool, Lorena Danae Perez Lopez.
This is a reproduction under a reduced budget, **not** the paper's released checkpoint: it is 8.5 Spearman points below the paper on STS-B test (see [Limitations](#limitations)). The supervised companion model is [`pyrawn/simcse-bert-base-snli-sup`](https://huggingface.co/pyrawn/simcse-bert-base-snli-sup) (78.75 test) and is better on every metric below.
## Usage
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("pyrawn/simcse-bert-base-snli-unsup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."])
print(model.similarity(emb[0:1], emb[1:2])) # cosine similarity
```
Architecture: `Transformer` (BERT-base, max 512 tokens) β†’ `Pooling(cls)` (last-layer `[CLS]` hidden state, **without** the MLP used during training and without BERT's pretrained pooler) β†’ `Normalize`.
## Training data
Not the paper's data. The paper trains unsupervised SimCSE on 10⁢ sentences sampled from English Wikipedia; this model was trained on **165,529 unique sentences** (premises and hypotheses, exact-string deduplication) extracted from a fixed 100k-record subset of the SNLI training set (`data/snli_train_100k.jsonl`, SHA-256 `0927a0b1…7f88`). Counts and checksums are in the project's `data/processed/MANIFEST.json`. SNLI sentences are short English image captions, a much narrower domain than Wikipedia. Labels are not used.
## Training recipe
From `runs/unsup_seed42_bs64/config.json`:
| Hyperparameter | Value |
|---|---|
| Mode | unsupervised (dropout noise as the only augmentation; in-batch negatives) |
| Base model | `bert-base-uncased` |
| Training pooling | `[CLS]` + MLP (`pooling: cls`, `mlp_only_train: true`) |
| Inference / eval pooling | `cls_before_pooler` (`[CLS]` without the MLP) |
| Batch size | 64 |
| Learning rate | 3e-5, linear decay, warmup ratio 0.0 |
| Optimizer | AdamW, weight decay 0.0, max grad norm 1.0 |
| Epochs | 1 (2,586 steps) |
| Temperature Ο„ | 0.05 |
| Max sequence length (training) | 32 |
| Dropout (hidden / attention) | 0.1 / 0.1 |
| Same dropout mask for both views | false |
| Precision | fp16 (AMP) |
| Seed | 42 |
| Dev evaluation / checkpoint selection | every 125 steps on STS-B dev; best kept (**step 125**) |
| Hardware | 1Γ— Tesla T4 (Kaggle), ~10 min |
| Libraries | torch 2.10.0, transformers 5.18.0, Python 3.12 |
## Evaluation
STS-B (`sentence-transformers/stsb`), protocol of the SimCSE paper (Appendix B): embed each sentence independently, cosine similarity, Spearman ρ against gold scores, no STS-B fine-tuning. Spearman Γ—100:
| Model | Dev | Test | Alignment ↓ | Uniformity ↓ |
|---|---|---|---|---|
| **This model** | **76.92** | **68.33** | **0.305** | **βˆ’2.963** |
| Supervised companion (`simcse-bert-base-snli-sup`) | 82.05 | 78.75 | 0.220 | βˆ’3.362 |
| raw `bert-base-uncased` (mean pooling) | 59.31 | 47.29 | 0.195 | βˆ’1.650 |
| SBERT-2019 (`bert-base-nli-mean-tokens`) | 80.77 | 76.98 | 0.193 | βˆ’3.049 |
| SimCSE paper, unsup BERT-base (reported) | 82.50 | 76.85 | β€” | β€” |
- Dev is the best periodic dev score during training (used for checkpoint selection, so optimistically biased). Test was scored once on the kept checkpoint and **reproduced after publishing** by reloading this repo from the Hub into an empty cache (same value to floating-point precision).
- Alignment / uniformity (Wang & Isola, 2020) on STS-B dev, L2-normalized: alignment over the 264 dev pairs with gold β‰₯ 4; uniformity over the 2,910 unique dev sentences.
- Paraphrase retrieval over STS-B dev (264 gold β‰₯ 4 pairs, 2,910-sentence corpus): Recall@1 0.830, Recall@5 0.958, MRR 0.885.
## Limitations
- **Gap to the paper: βˆ’5.58 dev / βˆ’8.52 test.** About four fifths of it is attributed to the training corpus (165k short SNLI captions instead of 10⁢ Wikipedia sentences). Dev peaked at the very first evaluation (step 125, 8,000 sentences seen) and then drifted down to ~74.6, so more training on this corpus would not help. The test gap is larger than the dev gap partly because the selected checkpoint is an isolated dev maximum (~1 point of dev-selection optimism). One configuration and one seed were run (no hyperparameter search); the single-seed noise floor is about Β±1 point.
- **Lexical-template failures.** Without paraphrase supervision the model keeps sentences with the same syntactic frame and topic close even when the action/object differ: "The man is stirring the rice." retrieves "The man is buttering the bread." (cosine 0.80, gold 0.4/5) above "A person is mixing a pot of rice."; "cracking eggs into a bowl" ranks the true paraphrase "broke raw eggs into a bowl" only 6th.
- **Same topic, different event.** News headlines sharing a template ("6.4-magnitude quake strikes off Indonesia" vs. "6.9-magnitude quake strikes off Russia's Kuril Islands", cosine 0.84, gold 1/5) are treated as near-paraphrases. The model encodes topic and form more than numbers or named entities.
- **Domain.** Trained only on English image-caption sentences of ≀ 32 tokens. Expect weaker results on long text, other languages, and specialized domains. Not evaluated on anything other than STS-B.
- Inherits the biases of BERT's pretraining data and of SNLI.
## Citation
```bibtex
@inproceedings{gao2021simcse,
title={SimCSE: Simple Contrastive Learning of Sentence Embeddings},
author={Gao, Tianyu and Yao, Xingcheng and Chen, Danqi},
booktitle={EMNLP},
year={2021}
}
```