Sentence Similarity
sentence-transformers
Safetensors
English
embedding_gemma2
feature-extraction
embeddings
retrieval
semantic-search
matryoshka
nasa
combustion
microgravity
spacecraft-fire-safety
Instructions to use rarfileexe/FlameLedger-Encoder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use rarfileexe/FlameLedger-Encoder with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("rarfileexe/FlameLedger-Encoder") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
|
Download README.md from rarfileexe/FlameLedger-Encoder: direct link, hf CLI and curl.
- Browser
- Download file 15.6 kB
-
https://huggingface.co/rarfileexe/FlameLedger-Encoder/resolve/main/README.md
- Command line
-
hf download hf://rarfileexe/FlameLedger-Encoder/README.md
-
curl -L -o README.md https://huggingface.co/rarfileexe/FlameLedger-Encoder/resolve/main/README.md
15.6 kB
| language: | |
| - en | |
| license: apache-2.0 | |
| library_name: sentence-transformers | |
| base_model: google/embeddinggemma-2 | |
| pipeline_tag: sentence-similarity | |
| tags: | |
| - sentence-transformers | |
| - sentence-similarity | |
| - feature-extraction | |
| - embeddings | |
| - retrieval | |
| - semantic-search | |
| - matryoshka | |
| - nasa | |
| - combustion | |
| - microgravity | |
| - spacecraft-fire-safety | |
| # FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval | |
| A text embedding model for finding passages in NASA microgravity-combustion and spacecraft fire-safety literature. It is [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) with its text encoder fine-tuned on synthetic search queries over public NASA Technical Reports Server (NTRS) records. | |
| Built by **Team Qubit** for the NASA Space Apps Challenge 2026 (*Flame in Freefall*) and used in production by [FlameLedger](https://flameledger.agenticerastudio.tech), an evidence research console that ranks NASA/NIST combustion experiments against a proposed cabin environment. | |
| - **Live demo:** https://flameledger.agenticerastudio.tech | |
| - **Code:** https://github.com/Muhammad-Hassan12/FlameLedger | |
| - **Embedding size:** 768 (Matryoshka: 768 / 512 / 256 / 128) | |
| - **Language:** English | |
| - **License:** Apache 2.0 (see [License and attribution](#license-and-attribution)) | |
| > **Read this first.** This is a small, domain-specific fine-tune (1,025 training triplets, 486 passages). Gains over the base model are measured on teacher-written synthetic queries and are modest. On a small teacher-drafted question set (n = 23, unreviewed) no improvement was detectable. BM25 is a strong baseline on this corpus, and this model is intended to be used *together with* BM25. See [Evaluation](#evaluation) and [Limitations](#limitations). | |
| --- | |
| ## Model details | |
| | | | | |
| |---|---| | |
| | **Base model** | `google/embeddinggemma-2` (Google DeepMind, 740M total parameters; 270M text backbone) | | |
| | **What was fine-tuned** | Text tower only (about 271M trainable parameters). Vision and audio towers were frozen and are unused. | | |
| | **This repository** | Text-only encoder export (FP32). Image and audio inputs are not supported. | | |
| | **Output** | 768-dimensional embeddings, truncatable to 512 / 256 / 128 (Matryoshka) | | |
| | **Similarity** | Cosine similarity (L2-normalize embeddings) | | |
| | **Max sequence length used in training** | 512 tokens (base model supports up to 8K) | | |
| | **Precision** | Use FP32 or BF16. The base model card warns against FP16. | | |
| | **Developed by** | Syed Muhammad Hassan (Team Qubit)| | |
| --- | |
| ## Intended use | |
| **Intended:** | |
| - Semantic and hybrid (dense + BM25) retrieval over NASA combustion, microgravity, and spacecraft fire-safety technical reports. | |
| - First-stage candidate retrieval in a RAG or research-assistant pipeline where results are shown with their sources. | |
| **Not intended:** | |
| - Fire-safety decision-making, material qualification, or any safety-critical judgment. Retrieval relevance is not scientific correctness, and the system must not be used as a substitute for expert review. | |
| - General-purpose or multilingual embedding. Use the base model for that. | |
| - Answer verification. This model ranks passages. It does not check claims. | |
| --- | |
| ## How to use | |
| ### Prefixes are required | |
| The model was trained with fixed text prefixes. **Use exactly these formats for queries and documents**, or retrieval quality will silently degrade. | |
| ```python | |
| QUERY_PREFIX = "task: search result | query: " | |
| def fmt_query(q: str) -> str: | |
| return QUERY_PREFIX + " ".join(str(q).split()) | |
| def fmt_doc(title: str, text: str) -> str: | |
| t = " ".join((title or "").split()) or "none" | |
| return f"title: {t} | text: {str(text).strip()}" | |
| ``` | |
| ### Sentence Transformers | |
| ```python | |
| from sentence_transformers import SentenceTransformer | |
| model = SentenceTransformer("YOUR_HF_USERNAME/flameledger-embeddinggemma2-nasa-combustion") | |
| model.max_seq_length = 512 | |
| queries = [fmt_query("At what oxygen concentration does flame spread stop in low-speed opposed flow?")] | |
| docs = [ | |
| fmt_doc("Combustion of solid fuel in very low speed oxygen streams", | |
| "Flame spread over thin solid fuel in quiescent and low-speed flows ..."), | |
| fmt_doc("Particle and Smoke Detection on ISS for Next Generation Smoke Detectors", | |
| "Smoke detector response depends on particle size distribution ..."), | |
| ] | |
| # Prefixes are added manually above. Use .encode() so no extra prompt is prepended. | |
| q = model.encode(queries, normalize_embeddings=True) | |
| d = model.encode(docs, normalize_embeddings=True) | |
| scores = q @ d.T # cosine similarity | |
| print(scores) | |
| ``` | |
| ### Using a smaller dimension (Matryoshka) | |
| ```python | |
| import numpy as np | |
| def truncate(emb, dim=256): | |
| emb = emb[:, :dim] | |
| return emb / np.linalg.norm(emb, axis=1, keepdims=True) | |
| q256, d256 = truncate(q, 256), truncate(d, 256) | |
| ``` | |
| Re-normalize after truncating. See the [dimension results](#effect-of-embedding-dimension) before going below 256. | |
| ### Hybrid retrieval (recommended) | |
| The best configuration in our evaluation fuses dense and lexical rankings with Reciprocal Rank Fusion: | |
| - Dense: cosine similarity at 768 dimensions. | |
| - Lexical: BM25 (k1 = 1.5, b = 0.75) over `title + " " + text`, tokenized as lowercase alphanumerics (keeping internal `.` and `-`, e.g. `6061-t6`). | |
| - Fusion: RRF with k = 60 over the top 100 of each ranking. | |
| A reference implementation (`HybridRetriever`, `BM25`, `rrf_scores`) is in the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger). | |
| --- | |
| ## Training | |
| ### Data | |
| Training data is synthetic. No human-written training questions were used. | |
| | Stage | Count | | |
| |---|---| | |
| | NTRS records fetched | 416 | | |
| | On-topic papers after de-duplication and topic filter | 399 | | |
| | Passages (chunks) | 486 (399 abstracts + 87 body chunks from 10 full-text PDFs) | | |
| | Valid teacher queries after rule-based validation | 1,412 (train 1,128 / dev 134 / test 150) | | |
| | Training triplets after round-trip filter | 1,025 (dev: 134) | | |
| Most papers are represented by their abstract only, because scanned PDFs were skipped rather than guessed. | |
| **Query generation.** Gemma 4 12B instruction-tuned (`google/gemma-4-12B-it`) wrote three queries per passage in three styles: `keyword`, `question`, and `scenario` (an engineering need the passage helps with). Teacher output was kept only if it passed rule checks: | |
| - length limits per style, | |
| - no meta-references such as "the passage" or "this paper", | |
| - no verbatim copying (no 7-word run copied from the passage), | |
| - every multi-digit or decimal number in the query must appear in the passage (blocks invented numbers), | |
| - de-duplication. | |
| **Round-trip filter (train split only).** Queries for which the base model ranked the source passage outside its top 15 were dropped (kept 1,025 of 1,128, or 91%). | |
| **Hard negatives.** Three per query, mined with the base model from the same split only, never from the same paper, skipping near-duplicates (cosine ≥ 0.92) and candidates scoring within 5% of the positive, to reduce false negatives. | |
| **Leak-free split.** Train / dev / test are split **by paper** (321 / 39 / 39 papers), with papers sharing a normalized title kept in the same split. | |
| ### Procedure | |
| | | | | |
| |---|---| | |
| | **Loss** | `MatryoshkaLoss` (dims 768, 512, 256, 128) wrapping `CachedMultipleNegativesRankingLoss` | | |
| | **Epochs** | 3 (17 steps per epoch) | | |
| | **Batch size** | 64 (cached MNRL mini-batch 16) | | |
| | **Learning rate** | 2e-5, warmup ratio 0.1, weight decay 0.01 | | |
| | **Precision** | FP32 master weights with BF16 autocast | | |
| | **Batch sampler** | `NO_DUPLICATES` | | |
| | **Checkpoint selection** | Best dev nDCG@10 (evaluated every 8 steps) | | |
| | **Max sequence length** | 512 | | |
| | **Seed** | 42 | | |
| | **Hardware** | 1× NVIDIA L4 (24 GB), Google Colab | | |
| | **Software** | sentence-transformers 6.2.0, transformers 5.22.1, PyTorch 2.11.0 | | |
| Dev-set results before and after fine-tuning (dev was used to pick the checkpoint, so treat these as optimistic): | |
| | Dev metric | Base | Fine-tuned | | |
| |---|---|---| | |
| | nDCG@10 | 0.877 | 0.916 | | |
| | MRR@10 | 0.844 | 0.890 | | |
| | Accuracy@1 | 0.769 | 0.821 | | |
| | Accuracy@5 | 0.948 | 0.993 | | |
| --- | |
| ## Evaluation | |
| All numbers below are from the training notebook's scorecard. Recall@k is the fraction of queries whose labeled source passage appears in the top k results. Confidence intervals are 95% paired bootstrap. | |
| Full scorecard output from the training notebook: [`scorecard.md`](./scorecard.md). | |
| ### Evidence quality | |
| | Query set | n | Origin | Strength | | |
| |---|---|---|---| | |
| | Synthetic test | 150 | Teacher-written, same pipeline and style as training queries | Weakest. Optimistic. | | |
| | Teacher-drafted gold | 23 | Harder, less keyword-matching questions drafted by the teacher. **Not human-reviewed.** | Weak (small n, unreviewed) | | |
| | Human-written gold | 0 | none | Not available | | |
| **No human-written evaluation questions exist for this model.** The numbers measure retrieval on teacher-written queries and should not be read as deployed accuracy. | |
| ### Synthetic test, held-out papers only (n = 150) | |
| Candidates are the 51 passages from the 39 test papers. This is the cleanest setting, but it is easy: scores saturate. | |
| | Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 | | |
| |---|---|---|---|---| | |
| | BM25 | 0.800 | 0.967 | 1.000 | 0.874 | | |
| | Base, dense 768 | 0.713 | 0.967 | 0.993 | 0.811 | | |
| | **Fine-tuned, dense 768** | 0.793 | 0.993 | 1.000 | 0.871 | | |
| | Base + BM25 (RRF) | 0.787 | 0.987 | 1.000 | 0.873 | | |
| | **Fine-tuned + BM25 (RRF)** | **0.813** | **1.000** | 1.000 | **0.891** | | |
| ### Synthetic test, full archive (n = 150) | |
| Candidates are all 486 passages. This includes passages from training papers that the fine-tuned model saw during training, so it is less clean than the table above and likely favors the fine-tuned model. | |
| | Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 | | |
| |---|---|---|---|---| | |
| | BM25 | 0.647 | 0.900 | 0.940 | 0.752 | | |
| | Base, dense 768 | 0.527 | 0.840 | 0.893 | 0.652 | | |
| | **Fine-tuned, dense 768** | 0.647 | 0.920 | 0.953 | 0.747 | | |
| | Base + BM25 (RRF) | 0.673 | 0.913 | 0.967 | 0.776 | | |
| | **Fine-tuned + BM25 (RRF)** | **0.713** | **0.940** | **0.987** | **0.807** | | |
| ### Teacher-drafted gold, unreviewed (n = 23) | |
| | Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 | | |
| |---|---|---|---|---| | |
| | BM25 | 0.304 | 0.609 | 0.739 | 0.436 | | |
| | Base, dense 768 | 0.391 | 0.609 | 0.739 | 0.506 | | |
| | Fine-tuned, dense 768 | 0.478 | 0.609 | 0.696 | 0.537 | | |
| | Base + BM25 (RRF) | 0.261 | 0.565 | 0.696 | 0.422 | | |
| | Fine-tuned + BM25 (RRF) | 0.391 | 0.696 | 0.696 | 0.505 | | |
| ### Fine-tuned minus base (95% paired bootstrap CI) | |
| | Query set | Comparison | Recall@5 | MRR@10 | | |
| |---|---|---|---| | |
| | Synthetic, held-out papers | dense 768 | +0.027 [+0.007, +0.060] | +0.060 [+0.031, +0.092] | | |
| | Synthetic, held-out papers | + BM25 (RRF) | +0.013 [+0.000, +0.033] | +0.018 [-0.002, +0.037] | | |
| | Synthetic, full archive | dense 768 | +0.080 [+0.033, +0.127] | +0.095 [+0.053, +0.141] | | |
| | Synthetic, full archive | + BM25 (RRF) | +0.027 [+0.000, +0.060] | +0.031 [-0.001, +0.065] | | |
| | Teacher-drafted gold (n = 23) | dense 768 | +0.000 [-0.130, +0.130] | +0.031 [-0.021, +0.099] | | |
| | Teacher-drafted gold (n = 23) | + BM25 (RRF) | +0.130 [+0.000, +0.261] | +0.083 [+0.014, +0.164] | | |
| ### How to read these results | |
| - Fine-tuning gives a **statistically supported improvement to dense retrieval** on synthetic queries (confidence intervals exclude zero for Recall@5 and MRR@10). | |
| - Once BM25 is added, the improvement over base + BM25 is **small**, and several intervals touch or cross zero. | |
| - On the harder drafted-gold set, dense retrieval shows **no detectable difference** (n = 23 is too small to conclude either way). | |
| - **BM25 alone is competitive**: on held-out synthetic queries its MRR@10 (0.874) matches the fine-tuned dense model (0.871). The practical recommendation is hybrid retrieval, not dense alone. | |
| - Synthetic queries share the teacher model's phrasing, so all synthetic scores are optimistic. | |
| ### Effect of embedding dimension | |
| Fine-tuned model, synthetic test (n = 150): | |
| | Dimension | Held-out Recall@5 | Held-out MRR@10 | Full-archive Recall@5 | Full-archive MRR@10 | | |
| |---|---|---|---|---| | |
| | 768 | 0.993 | 0.871 | 0.920 | 0.747 | | |
| | 512 | 0.987 | 0.880 | 0.907 | 0.752 | | |
| | 256 | 0.973 | 0.871 | 0.893 | 0.734 | | |
| | 128 | 0.967 | 0.865 | 0.833 | 0.720 | | |
| 768 and 512 are effectively equivalent here. 256 is a reasonable storage trade-off. 128 loses noticeable recall on the full archive, and the base model card also advises validating 128-d on your own workload. | |
| --- | |
| ## Limitations | |
| - **Very small training set.** 1,025 triplets from 486 passages (399 of them abstracts). The model has seen a narrow slice of the literature. | |
| - **Synthetic supervision and evaluation.** Training and test queries come from one teacher model. There is no human-written evaluation set, and the 23-question drafted set was not human-reviewed. | |
| - **Modest, partly uncertain gains.** See [How to read these results](#how-to-read-these-results). | |
| - **Narrow domain.** Trained only on NASA combustion and fire-safety text. Out-of-domain behavior was not evaluated and may be worse than the base model. | |
| - **English only.** Multilingual ability of the base model was not evaluated after fine-tuning. | |
| - **Abstract-heavy corpus.** Most papers contribute only their abstract, so retrieval of detailed in-body conditions (tables, run parameters) is largely untested. | |
| - **Sequence length.** Fine-tuned at 512 tokens. Long-document behavior was not evaluated. | |
| - **Text only.** The vision and audio towers were removed from this export. | |
| - **Retrieval is not correctness.** A relevant passage can still be misread, mismatched to a condition, or superseded. Always show sources and keep a human in the loop. | |
| ## Responsible use | |
| This model supports literature search. It is **not** a NASA product, is not endorsed by NASA or Google, and must not be used as a fire-safety decision system. | |
| ## AI and tool disclosure | |
| > We fine-tuned Google's EmbeddingGemma 2 to retrieve relevant NASA combustion research. Gemma 4 12B generated synthetic queries for training. FlameLedger combines embedding-based retrieval with BM25 keyword search. | |
| AI-assisted engineering tools (Codex, Antigravity) were used for software development and deployment. Gemini (`gemini-3.1-flash-lite`) is used in the FlameLedger application, not in this model, to draft cited briefings from retrieved evidence. | |
| ## License and attribution | |
| - This model is a derivative of `google/embeddinggemma-2`, released by Google DeepMind under Apache 2.0, and is released here under **Apache 2.0**. The upstream license page links Google's Gemma 4 license terms. Please review the [base model card](https://huggingface.co/google/embeddinggemma-2) for the terms that apply to the original weights. | |
| - Training queries were generated with `google/gemma-4-12B-it`. | |
| - The training corpus is derived from public NASA NTRS records. **No corpus text is redistributed with this model.** | |
| ## Citation | |
| ```bibtex | |
| @misc{flameledger_encoder_2026, | |
| title = {FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval}, | |
| author = {{Syed Muhammad Hassan}, {Team Qubit}}, | |
| year = {2026}, | |
| howpublished = {Hugging Face model repository}, | |
| note = {NASA Space Apps Challenge 2026, Flame in Freefall. Code: https://github.com/Muhammad-Hassan12/FlameLedger} | |
| } | |
| ``` | |
| Please also cite the base model, EmbeddingGemma 2 (Google DeepMind). | |
| ## Contact | |
| Open an issue on the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger). |