---
license: mit
datasets:
- microsoft/ms_marco
language:
- en
base_model:
- google-bert/bert-base-uncased
pipeline_tag: feature-extraction
library_name: transformers
tags:
- bert
- dense-retrieval
- semantic-search
- information-retrieval
- faiss
- retrieval
- sentence-transformers
- pytorch
- semantic
- embeddings
---


---
license: mit
language:
- en
library_name: transformers
pipeline_tag: feature-extraction
tags:
- bert
- dense-retrieval
- semantic-search
- information-retrieval
- ms-marco
- faiss
---

# BERT Dense Retriever

A dense semantic retrieval model fine-tuned on the **MS MARCO Passage Ranking** dataset using **BERT-base-uncased**.

The model encodes natural language queries and passages into dense vector embeddings that can be indexed with **FAISS** for efficient semantic search.

This repository contains the complete Hugging Face compatible model including tokenizer, configuration, and custom model implementation.

---

# Model Details

**Backbone**

- BERT-base-uncased

**Pooling**

- Mean Pooling

**Embedding Normalization**

- L2 Normalization

**Similarity Metric**

- Cosine Similarity

**Training Objective**

- CrossEntropy Loss over the similarity matrix (InfoNCE-style retrieval objective)

---

# Training Configuration

| Parameter | Value |
|-----------|-------|
| Optimizer | AdamW |
| Learning Rate | 2e-5 |
| Batch Size | 32 |
| Epochs | 10 |
| Weight Decay | 0.01 |
| Temperature | 0.05 |

---

# Evaluation Results

| Model | Recall@10 | MRR | nDCG@10 |
|--------|----------:|----:|---------:|
| BERT-base-uncased | 0.4793 | 0.2837 | 0.3301 |
| **Fine-tuned Dense Retriever** | **0.9693** | **0.8521** | **0.8810** |

The fine-tuned model substantially improves retrieval quality on the evaluation set compared with the untuned BERT-base encoder.

---


# 📊 BEIR Evaluation Results

The table below compares the proposed retrieval pipeline against the **BM25 baseline reported by the original BEIR benchmark**.

| Dataset | Retrieval Strategy | BM25 (BEIR) NDCG@10 | Pipeline NDCG@10 | Recall@10 | Recall@100 | Improvement |
|----------|-------------------|--------------------:|-----------------:|-----------:|------------:|------------:|
| FEVER | Hybrid + Cross Encoder | 0.7530 | **0.9791** | **0.9873** | **0.9937** | **+0.2261** |
| Quora | Dense + Cross Encoder | 0.7830 | **0.9686** | **0.9858** | **0.9938** | **+0.1856** |
| HotpotQA | Dense + Cross Encoder | 0.6030 | **0.8977** | **0.8853** | **0.8960** | **+0.2947** |
| FiQA | Dense + Cross Encoder | 0.2361 | **0.7512** | **0.8055** | — | **+0.5151** |
| TREC-COVID | Hybrid + Cross Encoder | 0.6559 | **0.6868** | **0.0181** | **0.1110** | **+0.0309** |

> **Note:** BM25 scores are taken from the original BEIR benchmark and are included as the lexical retrieval baseline for comparison.

---

# 🔍 Key Findings

- Strong zero-shot generalization across multiple retrieval domains.
- Significant improvements over the BEIR BM25 baseline on FEVER, Quora, HotpotQA, and FiQA.
- Hybrid retrieval (BM25 + Dense Retriever) improves retrieval quality for specialized biomedical documents in TREC-COVID.
- Cross-Encoder re-ranking substantially enhances the final ranking quality by leveraging full query-document interactions.

---

# Usage

```python
from transformers import AutoTokenizer, AutoModel

model_name = "Innovatewithapple/bert-dense-retriever"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
)

inputs = tokenizer(
    "What is deep learning?",
    return_tensors="pt"
)

embeddings = model(**inputs)

print(embeddings.shape)
```

---

# Intended Use

This model is designed for:

- Dense semantic retrieval
- Semantic search
- Question-passage retrieval
- Retrieval-Augmented Generation (RAG)
- Information retrieval research

---

# Source Code

GitHub Repository:

**https://github.com/Innovatewithapple/dense-semantic-retrieval**

---

# Author

**Mihir Vyas**