w831152001/TARS-S-BERT

This repository features an S-BERT model fine-tuned under an nn.Module evaluator.

By optimizing with a custom binary cross-entropy objective,

it effectively breaks the embedding anisotropy effect (cone effect) and aligns the original S-BERT representations with Word2Vec,

forcing the output logits to stretch dynamically across positive and negative spectrums.

---

This model yields a similarity score of 0.0011659159790724516 between 'I like to read.' and 'Reading is a good habit.',

compared to a score of 0.03844790533185005 from the vanilla S-BERT.

Note that lower scores indicate higher semantic similarity.

---

Because original S-BERT has anisotropy effect (cone effect),

the similarity score of any sentence is very close and cannot be distinguished. Not to mention transferring.

So I first trained an nn.Module network making the subtraction of embedding vector of one sentence in S-BERT and in Word2Vec with another similar sentence the closer the better.

My dataset is mteb/stsbenchmark-sts.

Then I used the nn.Module network evaluator to fine-tune S-BERT.

---

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("w831152001/TARS-S-BERT")

query_embedding = model.encode("I like to read.", convert_to_tensor=True)

corpus_embedding = model.encode("Reading is a good habit.", convert_to_tensor=True)

distance = torch.mean(torch.abs(corpus_embedding - query_embedding), dim=0).item()

print(distance)

sbert_model_base = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')

query_embedding_base = sbert_model_base.encode("I like to read.", convert_to_tensor=True)

corpus_embedding_base = sbert_model_base.encode("Reading is a good habit.", convert_to_tensor=True)

distance_base = torch.mean(torch.abs(corpus_embedding_base - query_embedding_base), dim=0).item()

print(distance_base)

---

The output of running the demo.py and it explains the transferring:

[input ticket]: 'I just bought a business-class flight ticket to Paris.'

 vs. [Five-star Luxury Hotel in Paris] -> distance diff: 0.0005596069968305528  
 
 vs. [Budget Hostel near Tokyo Station] -> distance diff: 0.0005826174165122211  
 
 vs. [Airport Capsule Hotel] -> distance diff: 0.0005628014914691448
 

=> The system automatically recommends the most suitable accommodation: [Five-star Luxury Hotel in Paris] (lowest distance diff: 0.0005596069968305528)

[input ticket]: 'My early flight departs tomorrow morning at 6 AM from the airport terminal.'

 vs. [Five-star Luxury Hotel in Paris] -> distance diff: 0.0005978584522381425  
 
 vs. [Budget Hostel near Tokyo Station] -> distance diff: 0.0005142632871866226  
 
 vs. [Airport Capsule Hotel] -> distance diff: 0.0004452229477465153
 

=> The system automatically recommends the most suitable accommodation: [Airport Capsule Hotel] (lowest distance diff: 0.0004452229477465153)