---
library_name: transformers
datasets:
- microsoft/ms_marco
license: mit
base_model:
- pranavupadhyaya52/rocky-embed
---


# RockyEmbed-Marco

 RockyEmbed-Marco is a lightweight, high-performance text embedding model built by contrastively fine-tuning RockyEmbed on the MS MARCO dataset. It is designed for efficient real-world retrieval tasks such as semantic search, RAG (Retrieval-Augmented Generation), and question answering systems.


---

### Overview

Modern embedding models often rely on large-scale architectures with billions of parameters. RockyEmbed-Marco takes a different approach:

Compact (~90M parameters)

Efficient (CPU-friendly inference)

Task-optimized (fine-tuned for retrieval)

Production-ready (designed for real-world RAG systems)


This model builds on RockyEmbed, which is pre-trained using distillation and optimized training strategies, and enhances it through contrastive learning on MS MARCO to improve retrieval quality.


---

### Model Architecture

Base Model: RockyEmbed

Parameters: ~90M

Embedding Dimension: (add your dimension here, e.g., 768 or 1024)

Training Strategy:

Stage 1: Distillation-based pretraining

Stage 2: Contrastive fine-tuning (MS MARCO)




---

### Training Details

Dataset

MS MARCO Passage Ranking Dataset

Large-scale dataset for training retrieval systems

Contains real-world queries and relevant passages



Objective Function

InfoNCE (Contrastive Loss)


The model learns to:

Pull semantically similar query-passage pairs closer

Push irrelevant pairs apart in embedding space



---

### Evaluation

RockyEmbed-Marco is evaluated on both benchmark datasets and real-world RAG scenarios:

MTEB Quora Subset (Massive Text Embedding Benchmark)(Quora)

Main Score: 0.64


RAGAS Evaluation Quora subset (RAG-specific metrics) (Quora)

Context Precision: 0.0583

Answer Correctness: 0.4717


These evaluations demonstrate:

Strong retrieval capability relative to model size

Practical effectiveness in downstream RAG pipelines



---

### Usage

Installation

pip install torch transformers sentence-transformers


---

Loading the Model

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("your-username/rockyembed-marco")

embeddings = model.encode([
    "What is contrastive learning?",
    "Explain retrieval augmented generation"
])


---

Example: Semantic Search

from sentence_transformers import util

query = "What is RAG?"
documents = [
    "RAG stands for Retrieval-Augmented Generation.",
    "Transformers are deep learning models.",
    "Contrastive learning improves embeddings."
]

query_emb = model.encode(query, convert_to_tensor=True)
doc_emb = model.encode(documents, convert_to_tensor=True)

scores = util.cos_sim(query_emb, doc_emb)

print(scores)


---

### Use Cases

🔍 Semantic Search

📚 Document Retrieval

🤖 Retrieval-Augmented Generation (RAG)

💬 Question Answering Systems

🧠 Embedding-based Clustering



---

### Design Philosophy

RockyEmbed-Marco is built with the following principles:

Efficiency over scale → smaller models, competitive performance

Practicality → optimized for real-world pipelines

Stability → improved training techniques to avoid gradient issues

Accessibility → usable on limited hardware (CPU-friendly)



---

### Key Insights

Contrastive fine-tuning significantly improves retrieval quality

Smaller models can compete with larger ones when trained effectively

Evaluation on RAG tasks is essential—not just benchmarks



---

### Future Work

Multi-domain fine-tuning

Hard-negative mining improvements

Multilingual support

Integration with lightweight LLM pipelines



---

### Contributing

Contributions, ideas, and improvements are welcome. Feel free to open issues or submit pull requests.


---

### License

(Add your license here, e.g., MIT / Apache 2.0)


---

### Contact

Pranav Upadhyaya
📧 pranavupadhyaya52@gmail.com
🔗 ORCID: https://orcid.org/0009-0008-8887-4349


---