FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval

A text embedding model for finding passages in NASA microgravity-combustion and spacecraft fire-safety literature. It is google/embeddinggemma-2 with its text encoder fine-tuned on synthetic search queries over public NASA Technical Reports Server (NTRS) records.

Built by Team Qubit for the NASA Space Apps Challenge 2026 (Flame in Freefall) and used in production by FlameLedger, an evidence research console that ranks NASA/NIST combustion experiments against a proposed cabin environment.

Read this first. This is a small, domain-specific fine-tune (1,025 training triplets, 486 passages). Gains over the base model are measured on teacher-written synthetic queries and are modest. On a small teacher-drafted question set (n = 23, unreviewed) no improvement was detectable. BM25 is a strong baseline on this corpus, and this model is intended to be used together with BM25. See Evaluation and Limitations.


Model details

Base model google/embeddinggemma-2 (Google DeepMind, 740M total parameters; 270M text backbone)
What was fine-tuned Text tower only (about 271M trainable parameters). Vision and audio towers were frozen and are unused.
This repository Text-only encoder export (FP32). Image and audio inputs are not supported.
Output 768-dimensional embeddings, truncatable to 512 / 256 / 128 (Matryoshka)
Similarity Cosine similarity (L2-normalize embeddings)
Max sequence length used in training 512 tokens (base model supports up to 8K)
Precision Use FP32 or BF16. The base model card warns against FP16.
Developed by Syed Muhammad Hassan (Team Qubit)

Intended use

Intended:

  • Semantic and hybrid (dense + BM25) retrieval over NASA combustion, microgravity, and spacecraft fire-safety technical reports.
  • First-stage candidate retrieval in a RAG or research-assistant pipeline where results are shown with their sources.

Not intended:

  • Fire-safety decision-making, material qualification, or any safety-critical judgment. Retrieval relevance is not scientific correctness, and the system must not be used as a substitute for expert review.
  • General-purpose or multilingual embedding. Use the base model for that.
  • Answer verification. This model ranks passages. It does not check claims.

How to use

Prefixes are required

The model was trained with fixed text prefixes. Use exactly these formats for queries and documents, or retrieval quality will silently degrade.

QUERY_PREFIX = "task: search result | query: "

def fmt_query(q: str) -> str:
    return QUERY_PREFIX + " ".join(str(q).split())

def fmt_doc(title: str, text: str) -> str:
    t = " ".join((title or "").split()) or "none"
    return f"title: {t} | text: {str(text).strip()}"

Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("YOUR_HF_USERNAME/flameledger-embeddinggemma2-nasa-combustion")
model.max_seq_length = 512

queries = [fmt_query("At what oxygen concentration does flame spread stop in low-speed opposed flow?")]
docs = [
    fmt_doc("Combustion of solid fuel in very low speed oxygen streams",
            "Flame spread over thin solid fuel in quiescent and low-speed flows ..."),
    fmt_doc("Particle and Smoke Detection on ISS for Next Generation Smoke Detectors",
            "Smoke detector response depends on particle size distribution ..."),
]

# Prefixes are added manually above. Use .encode() so no extra prompt is prepended.
q = model.encode(queries, normalize_embeddings=True)
d = model.encode(docs, normalize_embeddings=True)

scores = q @ d.T          # cosine similarity
print(scores)

Using a smaller dimension (Matryoshka)

import numpy as np

def truncate(emb, dim=256):
    emb = emb[:, :dim]
    return emb / np.linalg.norm(emb, axis=1, keepdims=True)

q256, d256 = truncate(q, 256), truncate(d, 256)

Re-normalize after truncating. See the dimension results before going below 256.

Hybrid retrieval (recommended)

The best configuration in our evaluation fuses dense and lexical rankings with Reciprocal Rank Fusion:

  • Dense: cosine similarity at 768 dimensions.
  • Lexical: BM25 (k1 = 1.5, b = 0.75) over title + " " + text, tokenized as lowercase alphanumerics (keeping internal . and -, e.g. 6061-t6).
  • Fusion: RRF with k = 60 over the top 100 of each ranking.

A reference implementation (HybridRetriever, BM25, rrf_scores) is in the FlameLedger repository.


Training

Data

Training data is synthetic. No human-written training questions were used.

Stage Count
NTRS records fetched 416
On-topic papers after de-duplication and topic filter 399
Passages (chunks) 486 (399 abstracts + 87 body chunks from 10 full-text PDFs)
Valid teacher queries after rule-based validation 1,412 (train 1,128 / dev 134 / test 150)
Training triplets after round-trip filter 1,025 (dev: 134)

Most papers are represented by their abstract only, because scanned PDFs were skipped rather than guessed.

Query generation. Gemma 4 12B instruction-tuned (google/gemma-4-12B-it) wrote three queries per passage in three styles: keyword, question, and scenario (an engineering need the passage helps with). Teacher output was kept only if it passed rule checks:

  • length limits per style,
  • no meta-references such as "the passage" or "this paper",
  • no verbatim copying (no 7-word run copied from the passage),
  • every multi-digit or decimal number in the query must appear in the passage (blocks invented numbers),
  • de-duplication.

Round-trip filter (train split only). Queries for which the base model ranked the source passage outside its top 15 were dropped (kept 1,025 of 1,128, or 91%).

Hard negatives. Three per query, mined with the base model from the same split only, never from the same paper, skipping near-duplicates (cosine โ‰ฅ 0.92) and candidates scoring within 5% of the positive, to reduce false negatives.

Leak-free split. Train / dev / test are split by paper (321 / 39 / 39 papers), with papers sharing a normalized title kept in the same split.

Procedure

Loss MatryoshkaLoss (dims 768, 512, 256, 128) wrapping CachedMultipleNegativesRankingLoss
Epochs 3 (17 steps per epoch)
Batch size 64 (cached MNRL mini-batch 16)
Learning rate 2e-5, warmup ratio 0.1, weight decay 0.01
Precision FP32 master weights with BF16 autocast
Batch sampler NO_DUPLICATES
Checkpoint selection Best dev nDCG@10 (evaluated every 8 steps)
Max sequence length 512
Seed 42
Hardware 1ร— NVIDIA L4 (24 GB), Google Colab
Software sentence-transformers 6.2.0, transformers 5.22.1, PyTorch 2.11.0

Dev-set results before and after fine-tuning (dev was used to pick the checkpoint, so treat these as optimistic):

Dev metric Base Fine-tuned
nDCG@10 0.877 0.916
MRR@10 0.844 0.890
Accuracy@1 0.769 0.821
Accuracy@5 0.948 0.993

Evaluation

All numbers below are from the training notebook's scorecard. Recall@k is the fraction of queries whose labeled source passage appears in the top k results. Confidence intervals are 95% paired bootstrap.

Full scorecard output from the training notebook: scorecard.md.

Evidence quality

Query set n Origin Strength
Synthetic test 150 Teacher-written, same pipeline and style as training queries Weakest. Optimistic.
Teacher-drafted gold 23 Harder, less keyword-matching questions drafted by the teacher. Not human-reviewed. Weak (small n, unreviewed)
Human-written gold 0 none Not available

No human-written evaluation questions exist for this model. The numbers measure retrieval on teacher-written queries and should not be read as deployed accuracy.

Synthetic test, held-out papers only (n = 150)

Candidates are the 51 passages from the 39 test papers. This is the cleanest setting, but it is easy: scores saturate.

Method Recall@1 Recall@5 Recall@10 MRR@10
BM25 0.800 0.967 1.000 0.874
Base, dense 768 0.713 0.967 0.993 0.811
Fine-tuned, dense 768 0.793 0.993 1.000 0.871
Base + BM25 (RRF) 0.787 0.987 1.000 0.873
Fine-tuned + BM25 (RRF) 0.813 1.000 1.000 0.891

Synthetic test, full archive (n = 150)

Candidates are all 486 passages. This includes passages from training papers that the fine-tuned model saw during training, so it is less clean than the table above and likely favors the fine-tuned model.

Method Recall@1 Recall@5 Recall@10 MRR@10
BM25 0.647 0.900 0.940 0.752
Base, dense 768 0.527 0.840 0.893 0.652
Fine-tuned, dense 768 0.647 0.920 0.953 0.747
Base + BM25 (RRF) 0.673 0.913 0.967 0.776
Fine-tuned + BM25 (RRF) 0.713 0.940 0.987 0.807

Teacher-drafted gold, unreviewed (n = 23)

Method Recall@1 Recall@5 Recall@10 MRR@10
BM25 0.304 0.609 0.739 0.436
Base, dense 768 0.391 0.609 0.739 0.506
Fine-tuned, dense 768 0.478 0.609 0.696 0.537
Base + BM25 (RRF) 0.261 0.565 0.696 0.422
Fine-tuned + BM25 (RRF) 0.391 0.696 0.696 0.505

Fine-tuned minus base (95% paired bootstrap CI)

Query set Comparison Recall@5 MRR@10
Synthetic, held-out papers dense 768 +0.027 [+0.007, +0.060] +0.060 [+0.031, +0.092]
Synthetic, held-out papers + BM25 (RRF) +0.013 [+0.000, +0.033] +0.018 [-0.002, +0.037]
Synthetic, full archive dense 768 +0.080 [+0.033, +0.127] +0.095 [+0.053, +0.141]
Synthetic, full archive + BM25 (RRF) +0.027 [+0.000, +0.060] +0.031 [-0.001, +0.065]
Teacher-drafted gold (n = 23) dense 768 +0.000 [-0.130, +0.130] +0.031 [-0.021, +0.099]
Teacher-drafted gold (n = 23) + BM25 (RRF) +0.130 [+0.000, +0.261] +0.083 [+0.014, +0.164]

How to read these results

  • Fine-tuning gives a statistically supported improvement to dense retrieval on synthetic queries (confidence intervals exclude zero for Recall@5 and MRR@10).
  • Once BM25 is added, the improvement over base + BM25 is small, and several intervals touch or cross zero.
  • On the harder drafted-gold set, dense retrieval shows no detectable difference (n = 23 is too small to conclude either way).
  • BM25 alone is competitive: on held-out synthetic queries its MRR@10 (0.874) matches the fine-tuned dense model (0.871). The practical recommendation is hybrid retrieval, not dense alone.
  • Synthetic queries share the teacher model's phrasing, so all synthetic scores are optimistic.

Effect of embedding dimension

Fine-tuned model, synthetic test (n = 150):

Dimension Held-out Recall@5 Held-out MRR@10 Full-archive Recall@5 Full-archive MRR@10
768 0.993 0.871 0.920 0.747
512 0.987 0.880 0.907 0.752
256 0.973 0.871 0.893 0.734
128 0.967 0.865 0.833 0.720

768 and 512 are effectively equivalent here. 256 is a reasonable storage trade-off. 128 loses noticeable recall on the full archive, and the base model card also advises validating 128-d on your own workload.


Limitations

  • Very small training set. 1,025 triplets from 486 passages (399 of them abstracts). The model has seen a narrow slice of the literature.
  • Synthetic supervision and evaluation. Training and test queries come from one teacher model. There is no human-written evaluation set, and the 23-question drafted set was not human-reviewed.
  • Modest, partly uncertain gains. See How to read these results.
  • Narrow domain. Trained only on NASA combustion and fire-safety text. Out-of-domain behavior was not evaluated and may be worse than the base model.
  • English only. Multilingual ability of the base model was not evaluated after fine-tuning.
  • Abstract-heavy corpus. Most papers contribute only their abstract, so retrieval of detailed in-body conditions (tables, run parameters) is largely untested.
  • Sequence length. Fine-tuned at 512 tokens. Long-document behavior was not evaluated.
  • Text only. The vision and audio towers were removed from this export.
  • Retrieval is not correctness. A relevant passage can still be misread, mismatched to a condition, or superseded. Always show sources and keep a human in the loop.

Responsible use

This model supports literature search. It is not a NASA product, is not endorsed by NASA or Google, and must not be used as a fire-safety decision system.

AI and tool disclosure

We fine-tuned Google's EmbeddingGemma 2 to retrieve relevant NASA combustion research. Gemma 4 12B generated synthetic queries for training. FlameLedger combines embedding-based retrieval with BM25 keyword search.

AI-assisted engineering tools (Codex, Antigravity) were used for software development and deployment. Gemini (gemini-3.1-flash-lite) is used in the FlameLedger application, not in this model, to draft cited briefings from retrieved evidence.

License and attribution

  • This model is a derivative of google/embeddinggemma-2, released by Google DeepMind under Apache 2.0, and is released here under Apache 2.0. The upstream license page links Google's Gemma 4 license terms. Please review the base model card for the terms that apply to the original weights.
  • Training queries were generated with google/gemma-4-12B-it.
  • The training corpus is derived from public NASA NTRS records. No corpus text is redistributed with this model.

Citation

@misc{flameledger_encoder_2026,
  title        = {FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval},
  author       = {{Syed Muhammad Hassan}, {Team Qubit}},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  note         = {NASA Space Apps Challenge 2026, Flame in Freefall. Code: https://github.com/Muhammad-Hassan12/FlameLedger}
}

Please also cite the base model, EmbeddingGemma 2 (Google DeepMind).

Contact

Open an issue on the FlameLedger repository.

Downloads last month
14
Safetensors
Model size
0.7B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for rarfileexe/FlameLedger-Encoder

Finetuned
(181)
this model