--- language: - en license: apache-2.0 library_name: sentence-transformers base_model: google/embeddinggemma-2 pipeline_tag: sentence-similarity tags: - sentence-transformers - sentence-similarity - feature-extraction - embeddings - retrieval - semantic-search - matryoshka - nasa - combustion - microgravity - spacecraft-fire-safety --- # FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval A text embedding model for finding passages in NASA microgravity-combustion and spacecraft fire-safety literature. It is [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) with its text encoder fine-tuned on synthetic search queries over public NASA Technical Reports Server (NTRS) records. Built by **Team Qubit** for the NASA Space Apps Challenge 2026 (*Flame in Freefall*) and used in production by [FlameLedger](https://flameledger.agenticerastudio.tech), an evidence research console that ranks NASA/NIST combustion experiments against a proposed cabin environment. - **Live demo:** https://flameledger.agenticerastudio.tech - **Code:** https://github.com/Muhammad-Hassan12/FlameLedger - **Embedding size:** 768 (Matryoshka: 768 / 512 / 256 / 128) - **Language:** English - **License:** Apache 2.0 (see [License and attribution](#license-and-attribution)) > **Read this first.** This is a small, domain-specific fine-tune (1,025 training triplets, 486 passages). Gains over the base model are measured on teacher-written synthetic queries and are modest. On a small teacher-drafted question set (n = 23, unreviewed) no improvement was detectable. BM25 is a strong baseline on this corpus, and this model is intended to be used *together with* BM25. See [Evaluation](#evaluation) and [Limitations](#limitations). --- ## Model details | | | |---|---| | **Base model** | `google/embeddinggemma-2` (Google DeepMind, 740M total parameters; 270M text backbone) | | **What was fine-tuned** | Text tower only (about 271M trainable parameters). Vision and audio towers were frozen and are unused. | | **This repository** | Text-only encoder export (FP32). Image and audio inputs are not supported. | | **Output** | 768-dimensional embeddings, truncatable to 512 / 256 / 128 (Matryoshka) | | **Similarity** | Cosine similarity (L2-normalize embeddings) | | **Max sequence length used in training** | 512 tokens (base model supports up to 8K) | | **Precision** | Use FP32 or BF16. The base model card warns against FP16. | | **Developed by** | Syed Muhammad Hassan (Team Qubit)| --- ## Intended use **Intended:** - Semantic and hybrid (dense + BM25) retrieval over NASA combustion, microgravity, and spacecraft fire-safety technical reports. - First-stage candidate retrieval in a RAG or research-assistant pipeline where results are shown with their sources. **Not intended:** - Fire-safety decision-making, material qualification, or any safety-critical judgment. Retrieval relevance is not scientific correctness, and the system must not be used as a substitute for expert review. - General-purpose or multilingual embedding. Use the base model for that. - Answer verification. This model ranks passages. It does not check claims. --- ## How to use ### Prefixes are required The model was trained with fixed text prefixes. **Use exactly these formats for queries and documents**, or retrieval quality will silently degrade. ```python QUERY_PREFIX = "task: search result | query: " def fmt_query(q: str) -> str: return QUERY_PREFIX + " ".join(str(q).split()) def fmt_doc(title: str, text: str) -> str: t = " ".join((title or "").split()) or "none" return f"title: {t} | text: {str(text).strip()}" ``` ### Sentence Transformers ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("YOUR_HF_USERNAME/flameledger-embeddinggemma2-nasa-combustion") model.max_seq_length = 512 queries = [fmt_query("At what oxygen concentration does flame spread stop in low-speed opposed flow?")] docs = [ fmt_doc("Combustion of solid fuel in very low speed oxygen streams", "Flame spread over thin solid fuel in quiescent and low-speed flows ..."), fmt_doc("Particle and Smoke Detection on ISS for Next Generation Smoke Detectors", "Smoke detector response depends on particle size distribution ..."), ] # Prefixes are added manually above. Use .encode() so no extra prompt is prepended. q = model.encode(queries, normalize_embeddings=True) d = model.encode(docs, normalize_embeddings=True) scores = q @ d.T # cosine similarity print(scores) ``` ### Using a smaller dimension (Matryoshka) ```python import numpy as np def truncate(emb, dim=256): emb = emb[:, :dim] return emb / np.linalg.norm(emb, axis=1, keepdims=True) q256, d256 = truncate(q, 256), truncate(d, 256) ``` Re-normalize after truncating. See the [dimension results](#effect-of-embedding-dimension) before going below 256. ### Hybrid retrieval (recommended) The best configuration in our evaluation fuses dense and lexical rankings with Reciprocal Rank Fusion: - Dense: cosine similarity at 768 dimensions. - Lexical: BM25 (k1 = 1.5, b = 0.75) over `title + " " + text`, tokenized as lowercase alphanumerics (keeping internal `.` and `-`, e.g. `6061-t6`). - Fusion: RRF with k = 60 over the top 100 of each ranking. A reference implementation (`HybridRetriever`, `BM25`, `rrf_scores`) is in the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger). --- ## Training ### Data Training data is synthetic. No human-written training questions were used. | Stage | Count | |---|---| | NTRS records fetched | 416 | | On-topic papers after de-duplication and topic filter | 399 | | Passages (chunks) | 486 (399 abstracts + 87 body chunks from 10 full-text PDFs) | | Valid teacher queries after rule-based validation | 1,412 (train 1,128 / dev 134 / test 150) | | Training triplets after round-trip filter | 1,025 (dev: 134) | Most papers are represented by their abstract only, because scanned PDFs were skipped rather than guessed. **Query generation.** Gemma 4 12B instruction-tuned (`google/gemma-4-12B-it`) wrote three queries per passage in three styles: `keyword`, `question`, and `scenario` (an engineering need the passage helps with). Teacher output was kept only if it passed rule checks: - length limits per style, - no meta-references such as "the passage" or "this paper", - no verbatim copying (no 7-word run copied from the passage), - every multi-digit or decimal number in the query must appear in the passage (blocks invented numbers), - de-duplication. **Round-trip filter (train split only).** Queries for which the base model ranked the source passage outside its top 15 were dropped (kept 1,025 of 1,128, or 91%). **Hard negatives.** Three per query, mined with the base model from the same split only, never from the same paper, skipping near-duplicates (cosine ≥ 0.92) and candidates scoring within 5% of the positive, to reduce false negatives. **Leak-free split.** Train / dev / test are split **by paper** (321 / 39 / 39 papers), with papers sharing a normalized title kept in the same split. ### Procedure | | | |---|---| | **Loss** | `MatryoshkaLoss` (dims 768, 512, 256, 128) wrapping `CachedMultipleNegativesRankingLoss` | | **Epochs** | 3 (17 steps per epoch) | | **Batch size** | 64 (cached MNRL mini-batch 16) | | **Learning rate** | 2e-5, warmup ratio 0.1, weight decay 0.01 | | **Precision** | FP32 master weights with BF16 autocast | | **Batch sampler** | `NO_DUPLICATES` | | **Checkpoint selection** | Best dev nDCG@10 (evaluated every 8 steps) | | **Max sequence length** | 512 | | **Seed** | 42 | | **Hardware** | 1× NVIDIA L4 (24 GB), Google Colab | | **Software** | sentence-transformers 6.2.0, transformers 5.22.1, PyTorch 2.11.0 | Dev-set results before and after fine-tuning (dev was used to pick the checkpoint, so treat these as optimistic): | Dev metric | Base | Fine-tuned | |---|---|---| | nDCG@10 | 0.877 | 0.916 | | MRR@10 | 0.844 | 0.890 | | Accuracy@1 | 0.769 | 0.821 | | Accuracy@5 | 0.948 | 0.993 | --- ## Evaluation All numbers below are from the training notebook's scorecard. Recall@k is the fraction of queries whose labeled source passage appears in the top k results. Confidence intervals are 95% paired bootstrap. Full scorecard output from the training notebook: [`scorecard.md`](./scorecard.md). ### Evidence quality | Query set | n | Origin | Strength | |---|---|---|---| | Synthetic test | 150 | Teacher-written, same pipeline and style as training queries | Weakest. Optimistic. | | Teacher-drafted gold | 23 | Harder, less keyword-matching questions drafted by the teacher. **Not human-reviewed.** | Weak (small n, unreviewed) | | Human-written gold | 0 | none | Not available | **No human-written evaluation questions exist for this model.** The numbers measure retrieval on teacher-written queries and should not be read as deployed accuracy. ### Synthetic test, held-out papers only (n = 150) Candidates are the 51 passages from the 39 test papers. This is the cleanest setting, but it is easy: scores saturate. | Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 | |---|---|---|---|---| | BM25 | 0.800 | 0.967 | 1.000 | 0.874 | | Base, dense 768 | 0.713 | 0.967 | 0.993 | 0.811 | | **Fine-tuned, dense 768** | 0.793 | 0.993 | 1.000 | 0.871 | | Base + BM25 (RRF) | 0.787 | 0.987 | 1.000 | 0.873 | | **Fine-tuned + BM25 (RRF)** | **0.813** | **1.000** | 1.000 | **0.891** | ### Synthetic test, full archive (n = 150) Candidates are all 486 passages. This includes passages from training papers that the fine-tuned model saw during training, so it is less clean than the table above and likely favors the fine-tuned model. | Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 | |---|---|---|---|---| | BM25 | 0.647 | 0.900 | 0.940 | 0.752 | | Base, dense 768 | 0.527 | 0.840 | 0.893 | 0.652 | | **Fine-tuned, dense 768** | 0.647 | 0.920 | 0.953 | 0.747 | | Base + BM25 (RRF) | 0.673 | 0.913 | 0.967 | 0.776 | | **Fine-tuned + BM25 (RRF)** | **0.713** | **0.940** | **0.987** | **0.807** | ### Teacher-drafted gold, unreviewed (n = 23) | Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 | |---|---|---|---|---| | BM25 | 0.304 | 0.609 | 0.739 | 0.436 | | Base, dense 768 | 0.391 | 0.609 | 0.739 | 0.506 | | Fine-tuned, dense 768 | 0.478 | 0.609 | 0.696 | 0.537 | | Base + BM25 (RRF) | 0.261 | 0.565 | 0.696 | 0.422 | | Fine-tuned + BM25 (RRF) | 0.391 | 0.696 | 0.696 | 0.505 | ### Fine-tuned minus base (95% paired bootstrap CI) | Query set | Comparison | Recall@5 | MRR@10 | |---|---|---|---| | Synthetic, held-out papers | dense 768 | +0.027 [+0.007, +0.060] | +0.060 [+0.031, +0.092] | | Synthetic, held-out papers | + BM25 (RRF) | +0.013 [+0.000, +0.033] | +0.018 [-0.002, +0.037] | | Synthetic, full archive | dense 768 | +0.080 [+0.033, +0.127] | +0.095 [+0.053, +0.141] | | Synthetic, full archive | + BM25 (RRF) | +0.027 [+0.000, +0.060] | +0.031 [-0.001, +0.065] | | Teacher-drafted gold (n = 23) | dense 768 | +0.000 [-0.130, +0.130] | +0.031 [-0.021, +0.099] | | Teacher-drafted gold (n = 23) | + BM25 (RRF) | +0.130 [+0.000, +0.261] | +0.083 [+0.014, +0.164] | ### How to read these results - Fine-tuning gives a **statistically supported improvement to dense retrieval** on synthetic queries (confidence intervals exclude zero for Recall@5 and MRR@10). - Once BM25 is added, the improvement over base + BM25 is **small**, and several intervals touch or cross zero. - On the harder drafted-gold set, dense retrieval shows **no detectable difference** (n = 23 is too small to conclude either way). - **BM25 alone is competitive**: on held-out synthetic queries its MRR@10 (0.874) matches the fine-tuned dense model (0.871). The practical recommendation is hybrid retrieval, not dense alone. - Synthetic queries share the teacher model's phrasing, so all synthetic scores are optimistic. ### Effect of embedding dimension Fine-tuned model, synthetic test (n = 150): | Dimension | Held-out Recall@5 | Held-out MRR@10 | Full-archive Recall@5 | Full-archive MRR@10 | |---|---|---|---|---| | 768 | 0.993 | 0.871 | 0.920 | 0.747 | | 512 | 0.987 | 0.880 | 0.907 | 0.752 | | 256 | 0.973 | 0.871 | 0.893 | 0.734 | | 128 | 0.967 | 0.865 | 0.833 | 0.720 | 768 and 512 are effectively equivalent here. 256 is a reasonable storage trade-off. 128 loses noticeable recall on the full archive, and the base model card also advises validating 128-d on your own workload. --- ## Limitations - **Very small training set.** 1,025 triplets from 486 passages (399 of them abstracts). The model has seen a narrow slice of the literature. - **Synthetic supervision and evaluation.** Training and test queries come from one teacher model. There is no human-written evaluation set, and the 23-question drafted set was not human-reviewed. - **Modest, partly uncertain gains.** See [How to read these results](#how-to-read-these-results). - **Narrow domain.** Trained only on NASA combustion and fire-safety text. Out-of-domain behavior was not evaluated and may be worse than the base model. - **English only.** Multilingual ability of the base model was not evaluated after fine-tuning. - **Abstract-heavy corpus.** Most papers contribute only their abstract, so retrieval of detailed in-body conditions (tables, run parameters) is largely untested. - **Sequence length.** Fine-tuned at 512 tokens. Long-document behavior was not evaluated. - **Text only.** The vision and audio towers were removed from this export. - **Retrieval is not correctness.** A relevant passage can still be misread, mismatched to a condition, or superseded. Always show sources and keep a human in the loop. ## Responsible use This model supports literature search. It is **not** a NASA product, is not endorsed by NASA or Google, and must not be used as a fire-safety decision system. ## AI and tool disclosure > We fine-tuned Google's EmbeddingGemma 2 to retrieve relevant NASA combustion research. Gemma 4 12B generated synthetic queries for training. FlameLedger combines embedding-based retrieval with BM25 keyword search. AI-assisted engineering tools (Codex, Antigravity) were used for software development and deployment. Gemini (`gemini-3.1-flash-lite`) is used in the FlameLedger application, not in this model, to draft cited briefings from retrieved evidence. ## License and attribution - This model is a derivative of `google/embeddinggemma-2`, released by Google DeepMind under Apache 2.0, and is released here under **Apache 2.0**. The upstream license page links Google's Gemma 4 license terms. Please review the [base model card](https://huggingface.co/google/embeddinggemma-2) for the terms that apply to the original weights. - Training queries were generated with `google/gemma-4-12B-it`. - The training corpus is derived from public NASA NTRS records. **No corpus text is redistributed with this model.** ## Citation ```bibtex @misc{flameledger_encoder_2026, title = {FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval}, author = {{Syed Muhammad Hassan}, {Team Qubit}}, year = {2026}, howpublished = {Hugging Face model repository}, note = {NASA Space Apps Challenge 2026, Flame in Freefall. Code: https://github.com/Muhammad-Hassan12/FlameLedger} } ``` Please also cite the base model, EmbeddingGemma 2 (Google DeepMind). ## Contact Open an issue on the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger).