chainuser1/Scripture_Genome_Multilingual_Head_BGE_M3

Scripture Genome — Multilingual Head for BGE-M3

Scripture Genome adds a scripture-specific representation to the multilingual understanding of BGE-M3. It maps a text embedding into 568 named features describing aspects of scripture, including themes, relationships, emotions, doctrine, literary form, and historical context.

This release is a trained head, not a standalone text encoder. It requires the BGE-M3 tokenizer and encoder. BGE-M3 was kept frozen: this project trained the additional head without changing the base model's parameters.

The selected release is question-hidden4096. It supports experimentation with scripture discovery across the Bible, Book of Mormon, Doctrine and Covenants, and Pearl of Great Price. It returns numerical features—not generated answers, scripture quotations, or authoritative interpretations.

Model at a glance

Property	Value
Required base model	BAAI/bge-m3
Pinned base revision	5617a9f61b028005a4858fdac845db406aefb181
Head architecture	Linear 1024 → 4096, GELU, Linear 4096 → 568, Sigmoid
Trainable head parameters	6,525,496
Input	1,024-dimensional, L2-normalized final-layer CLS embedding
Output	568 feature values in [0, 1], in gene_ids.txt order
Encoding length used in this project	512 tokens, including special tokens
Training/evaluation track	question
Question languages	English, Tagalog, Cebuano, Hebrew, Greek, Spanish, French, German
Head exports	FP32/FP16 Safetensors; weight-only INT8 Safetensors with an explicit loader; original NPZ ZIPs optional
Head license	MIT

The word gene refers to a named feature in the project's Scripture Genome schema. It has no biological meaning. These values are not calibrated probabilities, confidence scores, or measures of doctrinal truth.

How the pieces fit together

User text
   ↓
BGE-M3 tokenizer and frozen encoder
   ↓
L2-normalized CLS embedding: 1,024 values
   ├──→ BGE dense retrieval
   ↓
Scripture Genome head: 1024 → 4096 → 568
   ↓
Predicted scripture-answer profile
   ↓
Compare with stored verse genomes
   ↓
Optional fusion with BGE-M3 dense, sparse, and ColBERT signals
   ↓
Ranked verse IDs → scripture database → displayed passages

BGE-M3 supplies the general language representation. The head adds the project's scripture-specific feature space. A working search application also needs aligned verse IDs, verse genomes, scripture text, and any retrieval indexes it uses. The head weights do not contain those databases or indexes.

What was learned

Question labels were constructed from selected answer verses, rather than by directly assigning feature scores to question wording:

question_target = sum(answer_weight × stored_answer_verse_genome)

The weights for each question sum to one. Multiple answer verses can contribute, with more direct answers given greater weight. The source verse genomes already use the database's normalized [0, 1] representation; do not normalize them again as if they were raw 1–5 scores.

Consequently, a question's predicted genome should be understood as an expected profile of its selected scripture answers, not a literal annotation of the question itself. Historical or literary features can be predicted because the training answers share those characteristics, even when the question does not explicitly ask for them.

The labels and answer mappings are project-authored annotations, with AI-assisted scoring, question development, and translation. They reflect annotation choices and are not an interpretation-free theological ground truth.

Files and loading

Directly loadable Hugging Face release

The unpacked release provides the following files at repository root. The original ZIP downloads can remain as optional archives; they are not needed for this loading path.

File	Purpose
config.json	Custom head architecture, required base revision, checkpoint and gene-order identity
model.safetensors	FP32 head, directly loadable with the supplied Transformers model class
model.fp16.safetensors	FP16 weight-storage variant, preserving original FP32 biases
head_int8.safetensors	Original INT8 weights/scales/biases; load using inference.py, not generic automatic quantization
configuration_scripture_genome.py, modeling_scripture_genome.py	Auditable custom Transformers head definition
inference.py	Local loader for all three variants and a text-to-genome command
gene_ids.txt	Ordered 568 output-feature names
runtime_config.json, quantization_config.json	Retrieval configuration and explicit quantized-storage semantics
validation_q250.json, export_verification.json	Research validation and separate conversion checks
release_manifest.json	Checksums and conversion provenance
requirements.txt, LICENSE, README.md	Dependencies, MIT terms, and usage

No base-model weights, corpus databases, optimizer state, or retrieval indexes are included in this head-only release. Safetensors changes how weights are stored; it does not retrain or merge the head into BGE-M3. This is not GGUF, a chat model, or a hosted Inference Provider endpoint.

Recommended usage

Download the unpacked repository files into a local folder, then install dependencies:

python -m pip install -r requirements.txt
python inference.py --model-dir . --text "What is faith?"

The first text inference downloads the separate pinned BGE-M3 tokenizer/encoder. For an already extracted base, use --base-dir /path/to/bge-m3-pinned. That local base must be the pinned revision. No network access is needed with both model folders present. The helper returns dense embeddings' shape and genome values, not ranked verses.

Use --precision fp16 or --precision int8 to load the corresponding head storage variant. The helper performs head arithmetic in FP32 for portability; INT8 weights are dequantized at load time. These options do not quantize the base or promise faster inference.

For direct Transformers use:

import torch
from transformers import AutoModel, AutoTokenizer

# Local downloaded release folder, or the repository ID after files are uploaded.
HEAD = "./huggingface-question-hidden4096"
# HEAD = "chainuser1/Scripture_Genome_Multilingual_Head_BGE_M3"

# Review the supplied Python files before enabling custom-code execution.
# For a remote repository, also pin revision="<reviewed commit SHA>".
head = AutoModel.from_pretrained(HEAD, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(
    head.config.encoder_name, revision=head.config.encoder_revision,
)
base = AutoModel.from_pretrained(
    head.config.encoder_name, revision=head.config.encoder_revision,
).eval()

tokens = tokenizer(["What is faith?"], padding=True, truncation=False, return_tensors="pt")
if tokens["input_ids"].shape[1] > head.config.max_length:
    raise ValueError("Input exceeds the 512-token policy; shorten it explicitly")
with torch.inference_mode():
    hidden = base(**tokens).last_hidden_state
    dense = torch.nn.functional.normalize(hidden[:, 0, :].float(), dim=-1)
    genome = head(embeddings=dense).genome

print(dense.shape)   # [1, 1024]
print(genome.shape)  # [1, 568]

The head expects embeddings=..., not input_ids=.... Its custom class is not a replacement for the BGE-M3 encoder. Loading custom code requires explicit trust; Safetensors avoids pickled weight loading but does not make arbitrary Python code safe. No generic pipeline() integration is claimed.

variant="fp16" selects model.fp16.safetensors with from_pretrained(). Use an explicit supported computation dtype/device if you want half-precision execution, and cast the input accordingly. head_int8.safetensors is not a Transformers/bitsandbytes quantization format; use the bundled loader.

Original archive contents

The full and optimized ZIPs use the older filenames below. Extract them before using the legacy NPZ loader. They are retained for compatibility, not required alongside the unpacked Safetensors release.

File	Purpose
head_fp32.npz	Full-precision head weights; PyTorch state-dictionary key names
head_fp16.npz	Half-precision weights, with float32 biases
head_int8.npz	INT8 weights with per-output-row float32 scales and float32 biases
gene_ids.txt	Ordered feature names; one name per output position
head_card.json	Export metadata and, for the optimized bundle, quantization details
runtime_config.json	Encoder and retrieval configuration
validation_q250.json	The validation retrieval report summarized below
manifest.json	Bundle inventory and checksums
best.pt	Optional full training checkpoint, included in the full export

best.pt is not required for inference. The original NPZ files are custom exports, not Safetensors, ONNX, or a directly loadable Sentence Transformers model. Calling AutoModel.from_pretrained() on an old extracted NPZ-only ZIP folder will not construct this head; use the new unpacked release described above.

Legacy NPZ inference example

Install torch, numpy, and transformers in a suitable Python environment. Download/extract a head export and its gene_ids.txt. Set HEAD_DIR below to that folder. The example downloads the pinned BGE-M3 base separately; an internet connection is needed on its first run.

This example produces the genome vector only. It does not implement the full retrieval pipeline.

from pathlib import Path
import hashlib

import numpy as np
import torch
import torch.nn.functional as F
from torch import nn
from transformers import AutoModel, AutoTokenizer

BASE_MODEL = "BAAI/bge-m3"
BASE_REVISION = "5617a9f61b028005a4858fdac845db406aefb181"
HEAD_DIR = Path("./scripture-genome-question-hidden4096-full")
HEAD_FILE = "head_fp32.npz"  # Or head_fp16.npz / head_int8.npz.
GENE_IDS_SHA256 = "16878f92a2f6434b39ef6d68c633f2cd815751f2b4be8767a309e32442baf195"


def load_head(path):
    model = nn.Sequential(
        nn.Linear(1024, 4096), nn.GELU(),
        nn.Linear(4096, 568), nn.Sigmoid(),
    )
    state = {}
    with np.load(path, allow_pickle=False) as arrays:
        for exported_layer, torch_layer in ((0, 0), (1, 2)):
            if "0.weight" in arrays:  # Full FP32 export.
                weight = arrays[f"{torch_layer}.weight"].astype(np.float32)
                bias = arrays[f"{torch_layer}.bias"].astype(np.float32)
            else:  # Optimized FP16 or weight-only INT8 export.
                raw_weight = arrays[f"w{exported_layer}"]
                weight = raw_weight.astype(np.float32)
                if raw_weight.dtype == np.int8:
                    scale = arrays[f"scale{exported_layer}"].astype(np.float32)
                    weight = weight * scale[:, None]
                bias = arrays[f"b{exported_layer}"].astype(np.float32)
            if not (np.isfinite(weight).all() and np.isfinite(bias).all()):
                raise ValueError("Nonfinite head weights")
            state[f"{torch_layer}.weight"] = torch.from_numpy(weight)
            state[f"{torch_layer}.bias"] = torch.from_numpy(bias)
    model.load_state_dict(state, strict=True)
    return model.eval()


gene_path = HEAD_DIR / "gene_ids.txt"
if hashlib.sha256(gene_path.read_bytes()).hexdigest() != GENE_IDS_SHA256:
    raise ValueError("Unexpected gene order/file; verify the release bundle")
gene_ids = gene_path.read_text(encoding="utf-8").splitlines()
assert len(gene_ids) == 568

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, revision=BASE_REVISION)
encoder = AutoModel.from_pretrained(BASE_MODEL, revision=BASE_REVISION)
encoder = encoder.to(device).eval()
head = load_head(HEAD_DIR / HEAD_FILE).to(device)

texts = ["What is faith?", "Ano ang tunay na pananampalataya?"]
tokens = tokenizer(texts, padding=True, truncation=False, return_tensors="pt")
if tokens["input_ids"].shape[1] > 512:
    raise ValueError("Input exceeds this project's 512-token encoding policy")
tokens = {name: value.to(device) for name, value in tokens.items()}

with torch.inference_mode():
    hidden = encoder(**tokens).last_hidden_state
    dense = F.normalize(hidden[:, 0, :].float(), p=2, dim=-1)
    genomes = head(dense)

assert dense.shape == (len(texts), 1024)
assert genomes.shape == (len(texts), 568)
assert torch.isfinite(genomes).all()
print(genomes.cpu().numpy())

The example uses float32 computation for clarity and portability. The original cached embeddings used float16 computation/storage, so tiny numerical differences are expected. Recheck parity when changing runtime, pooling, precision, or tokenizer configuration. Do not substitute mean pooling, add a query instruction prefix, or sort the gene names.

The INT8 loader above dequantizes weights to float32. It demonstrates the file format and reduces download/storage size; it does not implement accelerated INT8 inference. Likewise, a quantized head does not quantize the much larger BGE-M3 base. Base-model quantization and mobile deployment require separate export and accuracy/performance checks. The packaged FP16 head has a limited parity probe; equivalent INT8 retrieval quality has not been established by the report below.

Retrieval: Experiment D

The project's experimental application default combines four signals:

1. Retrieve the top 200 candidates independently using dense similarity, sparse token matching, and weighted genome similarity.
2. Form their verse-ID union, containing at most 600 verses.
3. Compute BGE-M3 ColBERT token-level MaxSim scores for these candidates.
4. Standardize each of the four score arrays over that same candidate union and combine them:

final = 0.28 × z(dense)
      + 0.14 × z(sparse)
      + 0.28 × z(colbert)
      + 0.30 × z(weighted_genome)

Here z uses the candidate-set mean and sample standard deviation (ddof=1), with protection against zero variance. These final scores are ranking values, not probabilities or percentages. Candidate generation, standardization scope, and weights are part of the configuration—not interchangeable implementation details.

The released English retrieval configuration uses feature-family weights B=0.75, I=0.50, N=0.50, Q=0, T=0.75, and U=0.75, with all remaining families weighted 1.0. These are experimental retrieval choices, not changes to the trained head:

weighted_cosine(q, v) = cosine(sqrt(w) × q, sqrt(w) × v)

Apply the same feature weights to both query and verse vectors before L2 normalization. Family Q is excluded from this similarity calculation to reduce historical/provenance-driven matching; the head still outputs all 568 features.

BGE-M3's sparse and ColBERT representations come from the base model's own projection heads, not the Scripture Genome head. The minimal AutoModel example above obtains only dense embeddings. Experiment D also requires an implementation such as FlagEmbedding, the BGE-M3 sparse/ColBERT weights, and aligned passage representations. The current English retrieval bundle stores dense, genome, and sparse representations; ColBERT passage representations are computed for candidates on demand rather than supplied as a permanent token-vector index.

Multilingual support does not mean language has no effect on retrieval. In particular, sparse matching depends on shared token IDs. The reported experiment searches English passages with queries in eight languages; displaying a different translation afterward does not change that search into same-language retrieval. Optional exact-phrase search is a separate, language-specific text-index feature and does not require this model.

Training and data provenance

The packaged research corpus contains 47,400 question IDs with 379,200 question-language rows, and 41,995 verse genomes. Question translations retain their source question's target vector. Scripture and question records are joined to vectors by stable IDs, not assumed row positions across arbitrary files.

The head was trained on cached, frozen BGE-M3 embeddings. Training used eligible core-tier scripture rows and training-split questions across the eight question languages. Other scripture tiers were not automatically included.

The question track does not mean “questions only.” It assigns eligible verse records to training and holds out question IDs for validation/test. Translations of the same question share their split, with normalized duplicate grouping performed before language expansion. This tests new-question prediction against a known scripture corpus; it does not establish generalization to unseen chapters or unseen verse annotations.

Verified settings from the selected checkpoint:

Setting	Value
Optimizer	AdamW
Learning rate	0.001
Weight decay	0.01
Batch size	1,024
Samples per epoch	100,000, sampled with replacement
Sampling	Balanced over record-type/language groups
Loss	Mean squared error + 0.05 × mean cosine distance for nonzero targets
Gradient norm clipping	1.0
Seed	20261006
Selected checkpoint	End of epoch 193; 18,914 optimizer steps
Selection criterion	Lowest validation macro MAE
Stored validation macro MAE	0.0692111

An epoch here is a sampling budget, not one exhaustive pass over all records. The selected checkpoint need not be the final epoch of a longer training run.

The source corpus includes multiple scripture editions and project-generated annotations/translations. Inclusion in training does not certify a translation as official or expert-reviewed. Source-text rights remain separate from the head's license.

Evaluation

These are validation results, not a final held-out test result. The packaged retrieval report evaluates 250 question IDs in eight languages: 2,000 query-language rows against 41,995 English verse candidates.

Retrieval method	Recall@10	Recall@20	MRR@20	Weighted NDCG@20
BGE-M3 dense	0.0244	0.0378	0.0510	0.0311
Weighted genome	0.1351	0.1757	0.2417	0.1563
Weighted genome/dense 90:10	0.1449	0.1867	0.2603	0.1677
BGE dense + sparse + ColBERT	0.0315	0.0490	0.0627	0.0384
Experiment D: four signals	0.0670	0.0977	0.1368	0.0813

The 90:10 baseline fuses standardized genome and dense scores over the full candidate corpus; it is not a raw-cosine 90:10 average. Experiment D standardizes over its candidate union. BGE three-signal scoring in this experiment also uses that shared union, so it is not an independent end-to-end BGE benchmark.

Experiment D was retained for further application experimentation following qualitative inspection. It did not beat the 90:10 baseline on this annotated validation sample. Neither a few preferred examples nor these incomplete target labels alone establish overall user-facing superiority.

Additional report details:

• Candidate-union target recall: 0.4420, with a question-cluster bootstrap 95% interval of [0.4128, 0.4705].
• Mean measured search time: 2.142 seconds per query in the evaluation environment, not a mobile-device latency promise.
• The report uses the FP16 head export. Its small prediction-parity probe recorded maximum absolute error 0.000849 against FP32; this is not a full quantization benchmark.
• Bootstrap uncertainty treats question IDs as clusters. The eight translations are not 2,000 independent semantic questions.

Recall measures recovery of the project's listed answer targets. Unlisted but relevant verses can be penalized as non-targets. Only approximately 5,009 of the 41,995 verses appear as annotated answer targets, so retrieval evaluation covers a restricted part of the corpus. The labels also define the head's supervision; independently reviewed relevance judgments are needed for a broader assessment.

Limitations and responsible use

• Broad thematic similarity is not the same as a direct answer. A verse about the consequences of faith may outrank a verse defining faith.
• Weighted averages of multiple answers can blur distinctions between different aspects of a question.
• Historical, literary, and corpus-specific annotation patterns may influence predictions. Retrieval masking can reduce selected influences but cannot remove all learned bias.
• Language quality and retrieval performance vary. The base model's broader language coverage does not imply that this head is equally validated in every language.
• The model does not resolve theological disagreements or establish what a scriptural author definitively intended. Read results in context, with surrounding passages and edition information.
• Spiritual-support queries should not be presented as medical diagnosis or treatment advice. Scripture retrieval is not a substitute for professional care.
• Quantized exports, alternative runtimes, new scripture editions, changed gene schemas, and longer inputs require separate validation.
• No production mobile latency, security review, or universal retrieval-quality guarantee is claimed.

Reproducibility identifiers

Selected checkpoint SHA-256:
5a68ef73167f48899c927dfb5e25dfe8ff50d0b8e8310cd6f3ff3c93b8bf9952

Training dataset configuration hash:
ce3cf9db9a4c579b4dbe1e902ab6edd81a2067cd83901f88f29d0391eaf5e659

gene_ids.txt SHA-256:
16878f92a2f6434b39ef6d68c633f2cd815751f2b4be8767a309e32442baf195

Preserve the base revision, tokenizer, pooling, gene order, head identity, corpus/verse-ID mapping, and retrieval configuration together. Future training should create a new version and evaluation record rather than silently replace these artifacts.

License and acknowledgments

The Scripture Genome head is released under the MIT License; see LICENSE. BGE-M3 is a separate required dependency released under MIT by its upstream authors, as documented in its pinned model card. Preserve upstream license and attribution notices when redistributing base-model assets.

The head's MIT license does not relicense scripture editions, translation texts, or other third-party material. Review the rights for each dataset or application text bundle separately.

This project builds on the work of the BGE-M3 and FlagEmbedding authors:

• BGE-M3 model and documentation
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
• FlagEmbedding source code

When reporting results with this head, identify the release as Scripture Genome question-hidden4096, include its checkpoint hash, and specify the retrieval and precision configuration. Cite BGE-M3 separately; this head is not an official BAAI release.