File size: 5,860 Bytes
e1104ed | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 | ---
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
base_model: NeuML/hgbert-small
language: en
license: apache-2.0
---
# H.G. BERT Embeddings
This is a [H.G. BERT Small](https://hf.co/neuml/hgbert-small) model fined-tuned using [sentence-transformers](https://www.SBERT.net). It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search.
The training dataset was generating using a random sample of [Historical English Books](https://huggingface.co/datasets/NeuML/historical-english-books-similarity) title-abstract pairs and query-sentences pairs.
This model was trained using the following three step process.
- Train an initial model using [MultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss)
- [Mine hard negatives](https://www.sbert.net/docs/package_reference/util/hard_negatives.html) using the initial model
- Train a new model using the generated dataset with hard negatives
Unlike other models in this small domain model series, this model did not use distillation. This prevents against data leakage and learning modern similarity patterns. This keeps the model focused on 1899 and prior.
## Usage (txtai)
This model can be used to build embeddings databases with [txtai](https://github.com/neuml/txtai) for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).
```python
import txtai
embeddings = txtai.Embeddings(path="neuml/hgbert-small-embeddings", content=True)
embeddings.index(documents())
# Run a query
embeddings.search("query to run")
```
## Usage (Sentence-Transformers)
Alternatively, the model can be loaded with [sentence-transformers](https://www.SBERT.net).
```python
from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer("neuml/hgbert-small-embeddings")
embeddings = model.encode(sentences)
print(embeddings)
```
## Usage (Hugging Face Transformers)
The model can also be used directly with Transformers.
```python
from transformers import AutoTokenizer, AutoModel
import torch
# Mean Pooling - Take attention mask into account for correct averaging
def meanpooling(output, mask):
embeddings = output[0] # First element of model_output contains all token embeddings
mask = mask.unsqueeze(-1).expand(embeddings.size()).float()
return torch.sum(embeddings * mask, 1) / torch.clamp(mask.sum(1), min=1e-9)
# Sentences we want sentence embeddings for
sentences = ['This is an example sentence', 'Each sentence is converted']
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained("neuml/hgbert-small-embeddings")
model = AutoModel.from_pretrained("neuml/hgbert-small-embeddings")
# Tokenize sentences
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# Compute token embeddings
with torch.no_grad():
output = model(**inputs)
# Perform pooling. In this case, mean pooling.
embeddings = meanpooling(output, inputs['attention_mask'])
print("Sentence embeddings:")
print(embeddings)
```
## Evaluation Results
A [BEIR-compatible dataset](https://huggingface.co/datasets/NeuML/historical-english-books-similarity/tree/main/beir) was generated to facilitate the evaluation process.
Evaluation results are shown below. [NDCG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) is used as the evaluation metric.
| Model | Parameters | NDCG | Index Time | Search Time | Disk |
| ----------------------------------------------------------------------------------- | ---------- | --------- | ----------- | ----------- | --------- |
| [**H.G. BERT Small Embeddings**](https://hf.co/neuml/hgbert-small-embeddings) | **22.7M** | **50.12** | **3.82s** | **0.52s** | **23 MB** |
| [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2) | 22.7M | 44.82 | 4.50s | 0.60s | 23 MB |
| [DenseOn](https://hf.co/lightonai/DenseOn) | 149M | 47.06 | 17.52s | 1.16s | 46 MB |
| [EmbeddingGemma](https://hf.co/google/embeddinggemma-300m) | 300M | 47.60 | 24.47s | 2.11s | 31 MB |
| [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) | 600M | 45.75 | 27.60s | 2.98s | 62 MB |
| [Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) | 4000M | 48.21 | 129.18s | 15.17s | 154 MB |
| [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) | 8000M | 47.21 | 211.95s | 24.48s | 246 MB |
This model is a solid performer at a small size. It beats all models by a comfortable margin including ones with 8 billion parameters!
It can be used in CPU-only setups without trading off much on the accuracy front. It shows how small models can excel at specialized domains, requiring less compute and disk space.
## Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```
## More Information
Read more about the model in [this article](https://huggingface.co/blog/NeuML/hgbert-small).
|