Upload PawanEmbd model

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,161 @@
+---
+language: en
+license: apache-2.0
+tags:
+- sentence-transformers
+- sentence-similarity
+- embedding
+- knowledge-distillation
+datasets:
+- sentence-transformers/all-nli
+metrics:
+- cosine_similarity
+pipeline_tag: sentence-similarity
+---
+
+# PawanEmbd-68M
+
+A 68M parameter embedding model distilled from Granite-278M
+
+## Model Details
+
+- **Model Type**: Sentence Embedding Model
+- **Architecture**: Transformer-based encoder with projection layer
+- **Parameters**: ~5 million
+- **Teacher Model**: IBM Granite-278M Multilingual Embedding
+- **Training Method**: Knowledge Distillation
+- **Output Dimensions**: 768
+- **Max Sequence Length**: 512 tokens
+
+## Training Details
+
+This model was trained using knowledge distillation from the [IBM Granite-278M](https://huggingface.co/ibm-granite/granite-embedding-278m-multilingual) teacher model on the All-NLI dataset (SNLI + MultiNLI).
+
+### Training Hyperparameters
+
+- **Dataset**: sentence-transformers/all-nli (100K samples)
+- **Epochs**: 20
+- **Batch Size**: 32
+- **Learning Rate**: 5e-4 with OneCycleLR scheduler
+- **Loss Function**: Combined MSE + Cosine Similarity (α=0.5, β=0.5)
+- **Mixed Precision**: FP16 (AMP)
+- **Hardware**: NVIDIA T4 GPU
+
+
+## Usage
+
+### Using Transformers
+
+```
+from transformers import AutoModel, AutoTokenizer
+import torch
+import torch.nn.functional as F
+
+# Load model and tokenizer
+model = AutoModel.from_pretrained("dmedhi/pawanembd-68m")
+tokenizer = AutoTokenizer.from_pretrained("dmedhi/pawanembd-68m")
+
+# Encode sentences
+sentences = ["This is an example sentence", "Each sentence is converted to a vector"]
+encoded = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Get embeddings
+with torch.no_grad():
+    outputs = model(**encoded)
+    embeddings = outputs.pooler_output # Already normalized
+
+# Compute similarity
+similarity = F.cosine_similarity(embeddings[0:1], embeddings[1:2])
+print(f"Similarity: {similarity.item():.4f}")
+```
+
+
+### Using Sentence-Transformers
+
+```
+from sentence_transformers import SentenceTransformer
+from sentence_transformers.util import cos_sim
+
+model = SentenceTransformer("dmedhi/pawanembd-68m")
+
+sentences = ["This is an example sentence", "Each sentence is converted to a vector"]
+embeddings = model.encode(sentences)
+
+# Compute similarity
+similarity = cos_sim(embeddings, embeddings) 
+print(f"Similarity: {similarity.item():.4f}")
+```
+
+======================================================================
+COMPARING INFERENCE SPEED (Student vs Teacher)
+======================================================================
+Average inference time over 100 runs with 10 sentences (max_length=128):
+  Teacher Model: 17.944 ms
+  Student Model: 2.759 ms
+  Student is 6.5x faster than Teacher.
+
+  CPU speed comparision
+
+======================================================================
+COMPARING INFERENCE SPEED (Student vs Teacher)
+======================================================================
+Average inference time over 100 runs with 10 sentences (max_length=128):
+  Teacher Model: 269.578 ms
+  Student Model: 11.577 ms
+  Student is 23.3x faster than Teacher.
+
+## Performance
+
+### Comparison with Teacher Model
+
+| Metric | Teacher (Granite-278M) | Student (PawanEmbd-68M) |
+|--------|----------------------|----------------------|
+| Parameters | 278M | 68M (4.1x smaller) |
+| Model Size | ~1.1 GB | ~258.7 MB |
+| Inference Speed (CPU) | 269.57 ms | 11.57 (23.3x faster) |
+| Inference Speed (GPU) | 17.94.57 ms | 2.75 (6.5x faster) |
+| Cosine Similarity | 1.000 | 0.943 |
+
+
+## Intended Uses
+
+This model is suitable for:
+
+✅ **Semantic Search**: Find similar documents or passages
+✅ **Clustering**: Group similar texts together
+✅ **Duplicate Detection**: Identify near-duplicate content
+✅ **Recommendation Systems**: Find similar items
+✅ **Question Answering**: Retrieve relevant passages
+✅ **Sentence Similarity**: Measure semantic similarity between texts
+
+
+## Training Code
+
+The model was trained using PyTorch with knowledge distillation. Training code available at: TODO
+
+## Citation
+
+```
+@misc{pawanembdmodel2025,
+  author = {Dipankar Medhi},
+  title = {PawanEmbd: A Lightweight Embedding Model via Knowledge Distillation},
+  year = {2025},
+  publisher = {Hugging Face},
+  howpublished = { \url{https://huggingface.co/dmedhi/pawanembd-68m} }
+}
+```
+
+
+## Acknowledgments
+
+- Teacher model: [IBM Granite-278M](https://huggingface.co/ibm-granite/granite-embedding-278m-multilingual)
+- Training data: [Sentence-Transformers All-NLI](https://huggingface.co/datasets/sentence-transformers/all-nli)
+- Framework: Hugging Face Transformers & PyTorch
+
+## License
+
+Apache 2.0
+
+## Contact
+
+For questions or feedback, please open an issue on Github.

config.json

@@ -0,0 +1,19 @@
+{
+  "architectures": [
+    "PawanEmbdModel"
+  ],
+  "bos_token_id": 0,
+  "dropout": 0.1,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "hidden_size": 256,
+  "intermediate_size": 1024,
+  "max_position_embeddings": 512,
+  "model_type": "pawan_embd",
+  "num_heads": 4,
+  "num_layers": 4,
+  "output_size": 768,
+  "pad_token_id": 1,
+  "transformers_version": "4.57.2",
+  "vocab_size": 250002
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdc44ef043e813a4d5cadf7fe1b5d6f43a9f45d7d25ee4d8a632a3560a977daf
+size 271272400

requirements.txt

@@ -0,0 +1,3 @@
+transformers>=4.35.0
+torch>=2.0.0
+sentence-transformers>=2.2.0

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a56def25aa40facc030ea8b0b87f3688e4b3c39eb8b45d5702b3a1300fe2a20
+size 17082734

tokenizer_config.json

@@ -0,0 +1,55 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "250001": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "unk_token": "<unk>"
+}