File size: 1,435 Bytes
bf810b4 e301e0e bf810b4 489d90a b2c7114 489d90a b2c7114 489d90a bf810b4 fe73f95 bf810b4 fe73f95 bf810b4 fe73f95 bf810b4 073b066 bf810b4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
---
license: mit
datasets:
- cnmoro/LexicalTriplets
language:
- en
- pt
pipeline_tag: feature-extraction
library_name: sentence-transformers
---
This is a model trained on [cnmoro/LexicalTriplets](https://huggingface.co/datasets/cnmoro/LexicalTriplets) to produce lexical embeddings (not semantic!)
This can be used to compute lexical similarity between words or phrases.
Concept:
"Some text" will be similar to "Sm txt"
"King" will **not** be similar to "Queen" or "Royalty"
"Dog" will **not** be similar to "Animal"
"Doge" will be similar to "Dog"
```python
import torch, re, unicodedata
from transformers import AutoModel, AutoTokenizer
model_name = "cnmoro/LexicalEmbed-Base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)
model.eval()
def preprocess(text):
text = unicodedata.normalize('NFD', text)
text = ''.join(c for c in text if unicodedata.category(c) != 'Mn')
text = re.sub(r'[^\w\s]+', ' ', text.lower())
return re.sub(r'\s+', ' ', text).strip()
texts = ["hello world", "hel wor"]
texts = [ preprocess(s) for s in texts ]
inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
with torch.no_grad():
embeddings = model(**inputs)
cosine_sim = torch.nn.functional.cosine_similarity(embeddings[0], embeddings[1], dim=0)
print(f"Cosine Similarity: {cosine_sim.item()}") # 0.8966174125671387
``` |