File size: 4,100 Bytes
290235d 0244931 290235d 0244931 290235d 32f2951 290235d 0244931 290235d 988445b 290235d 988445b 290235d 988445b 290235d 988445b 290235d 558b455 290235d 0244931 290235d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 | ---
language: en
license: apache-2.0
tags:
- sentence-transformers
- sentence-similarity
- embedding
- knowledge-distillation
datasets:
- sentence-transformers/all-nli
metrics:
- cosine_similarity
pipeline_tag: sentence-similarity
---
# PawanEmbd-68M
A 68M parameter embedding model distilled from Granite-278M
## Model Details
- **Model Type**: Sentence Embedding Model
- **Architecture**: Transformer-based encoder with projection layer
- **Parameters**: ~68 million
- **Teacher Model**: IBM Granite-278M Multilingual Embedding
- **Training Method**: Knowledge Distillation
- **Output Dimensions**: 768
- **Max Sequence Length**: 512 tokens
## Training Details
This model was trained using knowledge distillation from the [IBM Granite-278M](https://huggingface.co/ibm-granite/granite-embedding-278m-multilingual) teacher model on the All-NLI dataset (SNLI + MultiNLI).
### Training Hyperparameters
- **Dataset**: sentence-transformers/all-nli (100K samples)
- **Epochs**: 20
- **Batch Size**: 32
- **Learning Rate**: 5e-4 with OneCycleLR scheduler
- **Loss Function**: Combined MSE + Cosine Similarity (α=0.5, β=0.5)
- **Mixed Precision**: FP16 (AMP)
- **Hardware**: NVIDIA T4 GPU
## Usage
### Using Transformers
```Python
from transformers import AutoModel, AutoTokenizer
import torch
import torch.nn.functional as F
# Load model and tokenizer
model = AutoModel.from_pretrained("dmedhi/PawanEmbd-68M")
tokenizer = AutoTokenizer.from_pretrained("dmedhi/PawanEmbd-68M")
# Encode sentences
sentences = ["This is an example sentence", "Each sentence is converted to a vector"]
encoded = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# Get embeddings
with torch.no_grad():
outputs = model(**encoded)
embeddings = outputs.pooler_output # Already normalized
# Compute similarity
similarity = F.cosine_similarity(embeddings[0:1], embeddings[1:2])
print(f"Similarity: {similarity.item():.4f}")
```
### Using Sentence-Transformers
```Python
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
# Load your model (should work now!)
model = SentenceTransformer("dmedhi/PawanEmbd-68M")
# Test encoding
sentences = ["This is an example sentence", "Each sentence is converted to a vector"]
embeddings = model.encode(sentences)
print(f"✅ Embeddings shape: {embeddings.shape}")
# Compute similarity
similarity = cos_sim(embeddings[0], embeddings[1])
print(f"✅ Similarity: {similarity.item():.4f}")
```
## Performance
### Comparison with Teacher Model
| Metric | Teacher (Granite-278M) | Student (PawanEmbd-68M) |
|--------|----------------------|----------------------|
| Parameters | 278M | 68M (4.1x smaller) |
| Model Size | ~1.1 GB | ~258.7 MB |
| Inference Speed (CPU) | 269.57 ms | 11.57 (23.3x faster) |
| Inference Speed (GPU) | 17.94.57 ms | 2.75 (6.5x faster) |
| Cosine Similarity | 1.000 | 0.943 |
## Intended Uses
This model is suitable for:
✅ **Semantic Search**: Find similar documents or passages \
✅ **Clustering**: Group similar texts together \
✅ **Duplicate Detection**: Identify near-duplicate content \
✅ **Recommendation Systems**: Find similar items \
✅ **Question Answering**: Retrieve relevant passages \
✅ **Sentence Similarity**: Measure semantic similarity between texts
## Training Code
The model was trained using PyTorch with knowledge distillation. Training code available at: TODO
## Citation
```
@misc{pawanembdmodel2025,
author = {Dipankar Medhi},
title = {PawanEmbd: A Lightweight Embedding Model via Knowledge Distillation},
year = {2025},
publisher = {Hugging Face},
howpublished = { \url{https://huggingface.co/dmedhi/PawanEmbd-68M} }
}
```
## Acknowledgments
- Teacher model: [IBM Granite-278M](https://huggingface.co/ibm-granite/granite-embedding-278m-multilingual)
- Training data: [Sentence-Transformers All-NLI](https://huggingface.co/datasets/sentence-transformers/all-nli)
- Framework: Hugging Face Transformers & PyTorch
## License
Apache 2.0
## Contact
For questions or feedback, please open an issue on Github.
|