Upgrade to MiniLM-L12-v2: NDCG@10 0.1831 (+9.3% over L6)

258fc23 verified 2 months ago

3.24 kB

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - bible
  - scripture
  - topical-search
license: mit
language: en
base_model: sentence-transformers/all-MiniLM-L12-v2
datasets:
  - LoveJesus/biblical-topical-dataset-chirho
metrics:
  - ndcg_at_10
  - mrr_at_10
  - map_at_10
model-index:
  - name: biblical-topical-search-chirho
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        metrics:
          - name: NDCG@10
            type: ndcg_at_10
            value: 0.1831
          - name: MRR@10
            type: mrr_at_10
            value: 0.1415
          - name: MAP@10
            type: map_at_10
            value: 0.1415
          - name: Accuracy@10
            type: accuracy_at_10
            value: 0.3188

Biblical Topical Search - Chirho

Semantic search model for finding biblical passages by topic. Fine-tuned on Nave's Topical Bible (30,000+ entries) and Treasury of Scripture Knowledge cross-references to retrieve relevant KJV verses for any theological query.

Model Details

Base Model: sentence-transformers/all-MiniLM-L12-v2 (33M params, 384-dim)
Training Data: 136,000 query-passage pairs from Nave's Topical Bible + TSK
Loss: MultipleNegativesRankingLoss
Training: 5 epochs, batch 16, lr 2e-5, cosine scheduler
Hardware: Apple M4 Pro (MPS), ~2 hours

Metrics (v2 - MiniLM-L12-v2 upgrade)

Metric	v1 (L6-v2)	v2 (L12-v2)	Change
NDCG@10	0.1676	0.1831	+9.3%
Accuracy@10	0.2882	0.3188	+10.6%
MRR@10	0.1305	0.1415	+8.4%
MAP@10	0.1305	0.1415	+8.4%

Training Trajectory (v2)

Epoch	NDCG@10	Accuracy@10
1	0.1514	0.257
2	0.1654	0.286
3	0.1749	0.302
4	0.1795	0.313
5	0.1831	0.319

Usage

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("LoveJesus/biblical-topical-search-chirho")

query = "What does the Bible say about forgiveness?"
verses = [
    "For if ye forgive men their trespasses, your heavenly Father will also forgive you. - Matthew 6:14",
    "In the beginning God created the heaven and the earth. - Genesis 1:1",
    "As far as the east is from the west, so far hath he removed our transgressions from us. - Psalm 103:12",
]

query_emb = model.encode([query])
verse_embs = model.encode(verses)

scores = np.dot(verse_embs, query_emb.T).flatten()
ranked = sorted(zip(scores, verses), reverse=True)
for score, verse in ranked:
    print(f"  {score:.3f}: {verse}")

Part of Bible.Systems

This model is part of the Bible.Systems AI models project — open-source machine learning for biblical scholarship.

Live Demo: HuggingFace Space
Dataset: LoveJesus/biblical-topical-dataset-chirho
All Models: LoveJesus on HuggingFace