Innovatewithapple/bert-dense-retriever

---

license: mit language: - en library_name: transformers pipeline_tag: feature-extraction tags: - bert - dense-retrieval - semantic-search - information-retrieval - ms-marco - faiss

BERT Dense Retriever

A dense semantic retrieval model fine-tuned on the MS MARCO Passage Ranking dataset using BERT-base-uncased.

The model encodes natural language queries and passages into dense vector embeddings that can be indexed with FAISS for efficient semantic search.

This repository contains the complete Hugging Face compatible model including tokenizer, configuration, and custom model implementation.

---

Model Details

Backbone

• BERT-base-uncased

Pooling

• Mean Pooling

Embedding Normalization

• L2 Normalization

Similarity Metric

• Cosine Similarity

Training Objective

• CrossEntropy Loss over the similarity matrix (InfoNCE-style retrieval objective)

---

Evaluation Results

Model	Recall@10	MRR	nDCG@10
BERT-base-uncased	0.4793	0.2837	0.3301
Fine-tuned Dense Retriever	0.9693	0.8521	0.8810

The fine-tuned model substantially improves retrieval quality on the evaluation set compared with the untuned BERT-base encoder.

---

Training Configuration

Parameter	Value
Optimizer	AdamW
Learning Rate	2e-5
Batch Size	32
Epochs	10
Weight Decay	0.01
Temperature	0.05

---

Usage

from transformers import AutoTokenizer
from modeling_bert_retriever import BertRetriever

model_name = "Innovatewithapple/bert-dense-retriever"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = BertRetriever.from_pretrained(
    model_name,
    trust_remote_code=True,
)

inputs = tokenizer(
    "What is deep learning?",
    return_tensors="pt"
)

embeddings = model(**inputs)

print(embeddings.shape)

---

Intended Use

This model is designed for:

• Dense semantic retrieval
• Semantic search
• Question-passage retrieval
• Retrieval-Augmented Generation (RAG)
• Information retrieval research

---

Source Code

GitHub Repository:

https://github.com/Innovatewithapple/dense-semantic-retrieval

---

Author

Mihir Vyas