README.md

---
license: mit
base_model: BAAI/bge-m3
library_name: transformers
language:
  - en
  - tl
  - ceb
  - he
  - el
  - es
  - fr
  - de
tags:
  - scripture
  - multilingual
  - information-retrieval
  - semantic-search
  - genome-regression
  - frozen-encoder
  - custom-head
---

# Scripture Genome — Multilingual Head for BGE-M3

Scripture Genome adds a scripture-specific representation to the multilingual understanding of **BGE-M3**. It maps a text embedding into 568 named features describing aspects of scripture, including themes, relationships, emotions, doctrine, literary form, and historical context.

**This release is a trained head, not a standalone text encoder. It requires the BGE-M3 tokenizer and encoder.** BGE-M3 was kept frozen: this project trained the additional head without changing the base model's parameters.

The selected release is **`question-hidden4096`**. It supports experimentation with scripture discovery across the Bible, Book of Mormon, Doctrine and Covenants, and Pearl of Great Price. It returns numerical features—not generated answers, scripture quotations, or authoritative interpretations.

## Model at a glance

| Property | Value |
| --- | --- |
| Required base model | [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3) |
| Pinned base revision | `5617a9f61b028005a4858fdac845db406aefb181` |
| Head architecture | Linear 1024 → 4096, GELU, Linear 4096 → 568, Sigmoid |
| Trainable head parameters | 6,525,496 |
| Input | 1,024-dimensional, L2-normalized final-layer CLS embedding |
| Output | 568 feature values in `[0, 1]`, in `gene_ids.txt` order |
| Encoding length used in this project | 512 tokens, including special tokens |
| Training/evaluation track | `question` |
| Question languages | English, Tagalog, Cebuano, Hebrew, Greek, Spanish, French, German |
| Head exports | FP32/FP16 Safetensors; weight-only INT8 Safetensors with an explicit loader; original NPZ ZIPs optional |
| Head license | MIT |

The word **gene** refers to a named feature in the project's Scripture Genome schema. It has no biological meaning. These values are not calibrated probabilities, confidence scores, or measures of doctrinal truth.

## How the pieces fit together

```text
User text
   ↓
BGE-M3 tokenizer and frozen encoder
   ↓
L2-normalized CLS embedding: 1,024 values
   ├──→ BGE dense retrieval
   ↓
Scripture Genome head: 1024 → 4096 → 568
   ↓
Predicted scripture-answer profile
   ↓
Compare with stored verse genomes
   ↓
Optional fusion with BGE-M3 dense, sparse, and ColBERT signals
   ↓
Ranked verse IDs → scripture database → displayed passages
```

BGE-M3 supplies the general language representation. The head adds the project's scripture-specific feature space. A working search application also needs aligned verse IDs, verse genomes, scripture text, and any retrieval indexes it uses. **The head weights do not contain those databases or indexes.**

## What was learned

Question labels were constructed from selected answer verses, rather than by directly assigning feature scores to question wording:

```text
question_target = sum(answer_weight × stored_answer_verse_genome)
```

The weights for each question sum to one. Multiple answer verses can contribute, with more direct answers given greater weight. The source verse genomes already use the database's normalized `[0, 1]` representation; do not normalize them again as if they were raw 1–5 scores.

Consequently, a question's predicted genome should be understood as an **expected profile of its selected scripture answers**, not a literal annotation of the question itself. Historical or literary features can be predicted because the training answers share those characteristics, even when the question does not explicitly ask for them.

The labels and answer mappings are project-authored annotations, with AI-assisted scoring, question development, and translation. They reflect annotation choices and are not an interpretation-free theological ground truth.

## Files and loading

### Directly loadable Hugging Face release

The unpacked release provides the following files at repository root. The original ZIP downloads can remain as optional archives; they are not needed for this loading path.

| File | Purpose |
| --- | --- |
| `config.json` | Custom head architecture, required base revision, checkpoint and gene-order identity |
| `model.safetensors` | FP32 head, directly loadable with the supplied Transformers model class |
| `model.fp16.safetensors` | FP16 weight-storage variant, preserving original FP32 biases |
| `head_int8.safetensors` | Original INT8 weights/scales/biases; load using `inference.py`, not generic automatic quantization |
| `configuration_scripture_genome.py`, `modeling_scripture_genome.py` | Auditable custom Transformers head definition |
| `inference.py` | Local loader for all three variants and a text-to-genome command |
| `gene_ids.txt` | Ordered 568 output-feature names |
| `runtime_config.json`, `quantization_config.json` | Retrieval configuration and explicit quantized-storage semantics |
| `validation_q250.json`, `export_verification.json` | Research validation and separate conversion checks |
| `release_manifest.json` | Checksums and conversion provenance |
| `requirements.txt`, `LICENSE`, `README.md` | Dependencies, MIT terms, and usage |

No base-model weights, corpus databases, optimizer state, or retrieval indexes are included in this head-only release. Safetensors changes how weights are stored; it does not retrain or merge the head into BGE-M3. This is not GGUF, a chat model, or a hosted Inference Provider endpoint.

### Recommended usage

Download the unpacked repository files into a local folder, then install dependencies:

```bash
python -m pip install -r requirements.txt
python inference.py --model-dir . --text "What is faith?"
```

The first text inference downloads the separate pinned BGE-M3 tokenizer/encoder. For an already extracted base, use `--base-dir /path/to/bge-m3-pinned`. That local base must be the pinned revision. No network access is needed with both model folders present. The helper returns dense embeddings' shape and genome values, not ranked verses.

Use `--precision fp16` or `--precision int8` to load the corresponding head storage variant. The helper performs head arithmetic in FP32 for portability; INT8 weights are dequantized at load time. These options do not quantize the base or promise faster inference.

For direct Transformers use:

```python
import torch
from transformers import AutoModel, AutoTokenizer

# Local downloaded release folder, or the repository ID after files are uploaded.
HEAD = "./huggingface-question-hidden4096"
# HEAD = "chainuser1/Scripture_Genome_Multilingual_Head_BGE_M3"

# Review the supplied Python files before enabling custom-code execution.
# For a remote repository, also pin revision="<reviewed commit SHA>".
head = AutoModel.from_pretrained(HEAD, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(
    head.config.encoder_name, revision=head.config.encoder_revision,
)
base = AutoModel.from_pretrained(
    head.config.encoder_name, revision=head.config.encoder_revision,
).eval()

tokens = tokenizer(["What is faith?"], padding=True, truncation=False, return_tensors="pt")
if tokens["input_ids"].shape[1] > head.config.max_length:
    raise ValueError("Input exceeds the 512-token policy; shorten it explicitly")
with torch.inference_mode():
    hidden = base(**tokens).last_hidden_state
    dense = torch.nn.functional.normalize(hidden[:, 0, :].float(), dim=-1)
    genome = head(embeddings=dense).genome

print(dense.shape)   # [1, 1024]
print(genome.shape)  # [1, 568]
```

The head expects `embeddings=...`, not `input_ids=...`. Its custom class is not a replacement for the BGE-M3 encoder. Loading custom code requires explicit trust; Safetensors avoids pickled weight loading but does not make arbitrary Python code safe. No generic `pipeline()` integration is claimed.

`variant="fp16"` selects `model.fp16.safetensors` with `from_pretrained()`. Use an explicit supported computation dtype/device if you want half-precision execution, and cast the input accordingly. `head_int8.safetensors` is **not** a Transformers/bitsandbytes quantization format; use the bundled loader.

### Original archive contents

The full and optimized ZIPs use the older filenames below. Extract them before using the legacy NPZ loader. They are retained for compatibility, not required alongside the unpacked Safetensors release.

| File | Purpose |
| --- | --- |
| `head_fp32.npz` | Full-precision head weights; PyTorch state-dictionary key names |
| `head_fp16.npz` | Half-precision weights, with float32 biases |
| `head_int8.npz` | INT8 weights with per-output-row float32 scales and float32 biases |
| `gene_ids.txt` | Ordered feature names; one name per output position |
| `head_card.json` | Export metadata and, for the optimized bundle, quantization details |
| `runtime_config.json` | Encoder and retrieval configuration |
| `validation_q250.json` | The validation retrieval report summarized below |
| `manifest.json` | Bundle inventory and checksums |
| `best.pt` | Optional full training checkpoint, included in the full export |

`best.pt` is not required for inference. The original NPZ files are custom exports, not Safetensors, ONNX, or a directly loadable Sentence Transformers model. Calling `AutoModel.from_pretrained()` on an old extracted NPZ-only ZIP folder will not construct this head; use the new unpacked release described above.

### Legacy NPZ inference example

Install `torch`, `numpy`, and `transformers` in a suitable Python environment. Download/extract a head export and its `gene_ids.txt`. Set `HEAD_DIR` below to that folder. The example downloads the pinned BGE-M3 base separately; an internet connection is needed on its first run.

This example produces the genome vector only. It does not implement the full retrieval pipeline.

```python
from pathlib import Path
import hashlib

import numpy as np
import torch
import torch.nn.functional as F
from torch import nn
from transformers import AutoModel, AutoTokenizer

BASE_MODEL = "BAAI/bge-m3"
BASE_REVISION = "5617a9f61b028005a4858fdac845db406aefb181"
HEAD_DIR = Path("./scripture-genome-question-hidden4096-full")
HEAD_FILE = "head_fp32.npz"  # Or head_fp16.npz / head_int8.npz.
GENE_IDS_SHA256 = "16878f92a2f6434b39ef6d68c633f2cd815751f2b4be8767a309e32442baf195"


def load_head(path):
    model = nn.Sequential(
        nn.Linear(1024, 4096), nn.GELU(),
        nn.Linear(4096, 568), nn.Sigmoid(),
    )
    state = {}
    with np.load(path, allow_pickle=False) as arrays:
        for exported_layer, torch_layer in ((0, 0), (1, 2)):
            if "0.weight" in arrays:  # Full FP32 export.
                weight = arrays[f"{torch_layer}.weight"].astype(np.float32)
                bias = arrays[f"{torch_layer}.bias"].astype(np.float32)
            else:  # Optimized FP16 or weight-only INT8 export.
                raw_weight = arrays[f"w{exported_layer}"]
                weight = raw_weight.astype(np.float32)
                if raw_weight.dtype == np.int8:
                    scale = arrays[f"scale{exported_layer}"].astype(np.float32)
                    weight = weight * scale[:, None]
                bias = arrays[f"b{exported_layer}"].astype(np.float32)
            if not (np.isfinite(weight).all() and np.isfinite(bias).all()):
                raise ValueError("Nonfinite head weights")
            state[f"{torch_layer}.weight"] = torch.from_numpy(weight)
            state[f"{torch_layer}.bias"] = torch.from_numpy(bias)
    model.load_state_dict(state, strict=True)
    return model.eval()


gene_path = HEAD_DIR / "gene_ids.txt"
if hashlib.sha256(gene_path.read_bytes()).hexdigest() != GENE_IDS_SHA256:
    raise ValueError("Unexpected gene order/file; verify the release bundle")
gene_ids = gene_path.read_text(encoding="utf-8").splitlines()
assert len(gene_ids) == 568

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, revision=BASE_REVISION)
encoder = AutoModel.from_pretrained(BASE_MODEL, revision=BASE_REVISION)
encoder = encoder.to(device).eval()
head = load_head(HEAD_DIR / HEAD_FILE).to(device)

texts = ["What is faith?", "Ano ang tunay na pananampalataya?"]
tokens = tokenizer(texts, padding=True, truncation=False, return_tensors="pt")
if tokens["input_ids"].shape[1] > 512:
    raise ValueError("Input exceeds this project's 512-token encoding policy")
tokens = {name: value.to(device) for name, value in tokens.items()}

with torch.inference_mode():
    hidden = encoder(**tokens).last_hidden_state
    dense = F.normalize(hidden[:, 0, :].float(), p=2, dim=-1)
    genomes = head(dense)

assert dense.shape == (len(texts), 1024)
assert genomes.shape == (len(texts), 568)
assert torch.isfinite(genomes).all()
print(genomes.cpu().numpy())
```

The example uses float32 computation for clarity and portability. The original cached embeddings used float16 computation/storage, so tiny numerical differences are expected. Recheck parity when changing runtime, pooling, precision, or tokenizer configuration. Do not substitute mean pooling, add a query instruction prefix, or sort the gene names.

The INT8 loader above **dequantizes weights to float32**. It demonstrates the file format and reduces download/storage size; it does not implement accelerated INT8 inference. Likewise, a quantized head does not quantize the much larger BGE-M3 base. Base-model quantization and mobile deployment require separate export and accuracy/performance checks. The packaged FP16 head has a limited parity probe; equivalent INT8 retrieval quality has not been established by the report below.

## Retrieval: Experiment D

The project's experimental application default combines four signals:

1. Retrieve the top 200 candidates independently using dense similarity, sparse token matching, and weighted genome similarity.
2. Form their verse-ID union, containing at most 600 verses.
3. Compute BGE-M3 ColBERT token-level MaxSim scores for these candidates.
4. Standardize each of the four score arrays over that same candidate union and combine them:

```text
final = 0.28 × z(dense)
      + 0.14 × z(sparse)
      + 0.28 × z(colbert)
      + 0.30 × z(weighted_genome)
```

Here `z` uses the candidate-set mean and sample standard deviation (`ddof=1`), with protection against zero variance. These final scores are ranking values, not probabilities or percentages. Candidate generation, standardization scope, and weights are part of the configuration—not interchangeable implementation details.

The released English retrieval configuration uses feature-family weights `B=0.75`, `I=0.50`, `N=0.50`, `Q=0`, `T=0.75`, and `U=0.75`, with all remaining families weighted `1.0`. These are experimental retrieval choices, not changes to the trained head:

```text
weighted_cosine(q, v) = cosine(sqrt(w) × q, sqrt(w) × v)
```

Apply the same feature weights to both query and verse vectors before L2 normalization. Family Q is excluded from this similarity calculation to reduce historical/provenance-driven matching; the head still outputs all 568 features.

BGE-M3's sparse and ColBERT representations come from the base model's own projection heads, **not** the Scripture Genome head. The minimal `AutoModel` example above obtains only dense embeddings. Experiment D also requires an implementation such as FlagEmbedding, the BGE-M3 sparse/ColBERT weights, and aligned passage representations. The current English retrieval bundle stores dense, genome, and sparse representations; ColBERT passage representations are computed for candidates on demand rather than supplied as a permanent token-vector index.

Multilingual support does not mean language has no effect on retrieval. In particular, sparse matching depends on shared token IDs. The reported experiment searches English passages with queries in eight languages; displaying a different translation afterward does not change that search into same-language retrieval. Optional exact-phrase search is a separate, language-specific text-index feature and does not require this model.

## Training and data provenance

The packaged research corpus contains 47,400 question IDs with 379,200 question-language rows, and 41,995 verse genomes. Question translations retain their source question's target vector. Scripture and question records are joined to vectors by stable IDs, not assumed row positions across arbitrary files.

The head was trained on cached, frozen BGE-M3 embeddings. Training used eligible core-tier scripture rows and training-split questions across the eight question languages. Other scripture tiers were not automatically included.

**The `question` track does not mean “questions only.”** It assigns eligible verse records to training and holds out question IDs for validation/test. Translations of the same question share their split, with normalized duplicate grouping performed before language expansion. This tests new-question prediction against a known scripture corpus; it does not establish generalization to unseen chapters or unseen verse annotations.

Verified settings from the selected checkpoint:

| Setting | Value |
| --- | --- |
| Optimizer | AdamW |
| Learning rate | 0.001 |
| Weight decay | 0.01 |
| Batch size | 1,024 |
| Samples per epoch | 100,000, sampled with replacement |
| Sampling | Balanced over record-type/language groups |
| Loss | Mean squared error + 0.05 × mean cosine distance for nonzero targets |
| Gradient norm clipping | 1.0 |
| Seed | 20261006 |
| Selected checkpoint | End of epoch 193; 18,914 optimizer steps |
| Selection criterion | Lowest validation macro MAE |
| Stored validation macro MAE | 0.0692111 |

An epoch here is a sampling budget, not one exhaustive pass over all records. The selected checkpoint need not be the final epoch of a longer training run.

The source corpus includes multiple scripture editions and project-generated annotations/translations. Inclusion in training does not certify a translation as official or expert-reviewed. Source-text rights remain separate from the head's license.

## Evaluation

These are **validation results, not a final held-out test result**. The packaged retrieval report evaluates 250 question IDs in eight languages: 2,000 query-language rows against 41,995 English verse candidates.

| Retrieval method | Recall@10 | Recall@20 | MRR@20 | Weighted NDCG@20 |
| --- | ---: | ---: | ---: | ---: |
| BGE-M3 dense | 0.0244 | 0.0378 | 0.0510 | 0.0311 |
| Weighted genome | 0.1351 | 0.1757 | 0.2417 | 0.1563 |
| Weighted genome/dense 90:10 | **0.1449** | **0.1867** | **0.2603** | **0.1677** |
| BGE dense + sparse + ColBERT | 0.0315 | 0.0490 | 0.0627 | 0.0384 |
| Experiment D: four signals | 0.0670 | 0.0977 | 0.1368 | 0.0813 |

The 90:10 baseline fuses standardized genome and dense scores over the full candidate corpus; it is not a raw-cosine 90:10 average. Experiment D standardizes over its candidate union. BGE three-signal scoring in this experiment also uses that shared union, so it is not an independent end-to-end BGE benchmark.

Experiment D was retained for further application experimentation following qualitative inspection. **It did not beat the 90:10 baseline on this annotated validation sample.** Neither a few preferred examples nor these incomplete target labels alone establish overall user-facing superiority.

Additional report details:

- Candidate-union target recall: **0.4420**, with a question-cluster bootstrap 95% interval of **[0.4128, 0.4705]**.
- Mean measured search time: **2.142 seconds per query** in the evaluation environment, not a mobile-device latency promise.
- The report uses the FP16 head export. Its small prediction-parity probe recorded maximum absolute error **0.000849** against FP32; this is not a full quantization benchmark.
- Bootstrap uncertainty treats question IDs as clusters. The eight translations are not 2,000 independent semantic questions.

Recall measures recovery of the project's listed answer targets. Unlisted but relevant verses can be penalized as non-targets. Only approximately 5,009 of the 41,995 verses appear as annotated answer targets, so retrieval evaluation covers a restricted part of the corpus. The labels also define the head's supervision; independently reviewed relevance judgments are needed for a broader assessment.

## Limitations and responsible use

- Broad thematic similarity is not the same as a direct answer. A verse about the consequences of faith may outrank a verse defining faith.
- Weighted averages of multiple answers can blur distinctions between different aspects of a question.
- Historical, literary, and corpus-specific annotation patterns may influence predictions. Retrieval masking can reduce selected influences but cannot remove all learned bias.
- Language quality and retrieval performance vary. The base model's broader language coverage does not imply that this head is equally validated in every language.
- The model does not resolve theological disagreements or establish what a scriptural author definitively intended. Read results in context, with surrounding passages and edition information.
- Spiritual-support queries should not be presented as medical diagnosis or treatment advice. Scripture retrieval is not a substitute for professional care.
- Quantized exports, alternative runtimes, new scripture editions, changed gene schemas, and longer inputs require separate validation.
- No production mobile latency, security review, or universal retrieval-quality guarantee is claimed.

## Reproducibility identifiers

```text
Selected checkpoint SHA-256:
5a68ef73167f48899c927dfb5e25dfe8ff50d0b8e8310cd6f3ff3c93b8bf9952

Training dataset configuration hash:
ce3cf9db9a4c579b4dbe1e902ab6edd81a2067cd83901f88f29d0391eaf5e659

gene_ids.txt SHA-256:
16878f92a2f6434b39ef6d68c633f2cd815751f2b4be8767a309e32442baf195
```

Preserve the base revision, tokenizer, pooling, gene order, head identity, corpus/verse-ID mapping, and retrieval configuration together. Future training should create a new version and evaluation record rather than silently replace these artifacts.

## License and acknowledgments

The Scripture Genome head is released under the **MIT License**; see [LICENSE](LICENSE). BGE-M3 is a separate required dependency released under MIT by its upstream authors, as documented in its [pinned model card](https://huggingface.co/BAAI/bge-m3/blob/5617a9f61b028005a4858fdac845db406aefb181/README.md). Preserve upstream license and attribution notices when redistributing base-model assets.

The head's MIT license does **not** relicense scripture editions, translation texts, or other third-party material. Review the rights for each dataset or application text bundle separately.

This project builds on the work of the BGE-M3 and FlagEmbedding authors:

- [BGE-M3 model and documentation](https://huggingface.co/BAAI/bge-m3)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://arxiv.org/abs/2402.03216)
- [FlagEmbedding source code](https://github.com/FlagOpen/FlagEmbedding)

When reporting results with this head, identify the release as **Scripture Genome `question-hidden4096`**, include its checkpoint hash, and specify the retrieval and precision configuration. Cite BGE-M3 separately; this head is not an official BAAI release.