FlameLedger-Encoder / README.md
rarfileexe's picture
Update README.md
cb66b20 verified
|
Raw History Blame Contribute Delete
15.6 kB
---
language:
- en
license: apache-2.0
library_name: sentence-transformers
base_model: google/embeddinggemma-2
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- embeddings
- retrieval
- semantic-search
- matryoshka
- nasa
- combustion
- microgravity
- spacecraft-fire-safety
---
# FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval
A text embedding model for finding passages in NASA microgravity-combustion and spacecraft fire-safety literature. It is [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) with its text encoder fine-tuned on synthetic search queries over public NASA Technical Reports Server (NTRS) records.
Built by **Team Qubit** for the NASA Space Apps Challenge 2026 (*Flame in Freefall*) and used in production by [FlameLedger](https://flameledger.agenticerastudio.tech), an evidence research console that ranks NASA/NIST combustion experiments against a proposed cabin environment.
- **Live demo:** https://flameledger.agenticerastudio.tech
- **Code:** https://github.com/Muhammad-Hassan12/FlameLedger
- **Embedding size:** 768 (Matryoshka: 768 / 512 / 256 / 128)
- **Language:** English
- **License:** Apache 2.0 (see [License and attribution](#license-and-attribution))
> **Read this first.** This is a small, domain-specific fine-tune (1,025 training triplets, 486 passages). Gains over the base model are measured on teacher-written synthetic queries and are modest. On a small teacher-drafted question set (n = 23, unreviewed) no improvement was detectable. BM25 is a strong baseline on this corpus, and this model is intended to be used *together with* BM25. See [Evaluation](#evaluation) and [Limitations](#limitations).
---
## Model details
| | |
|---|---|
| **Base model** | `google/embeddinggemma-2` (Google DeepMind, 740M total parameters; 270M text backbone) |
| **What was fine-tuned** | Text tower only (about 271M trainable parameters). Vision and audio towers were frozen and are unused. |
| **This repository** | Text-only encoder export (FP32). Image and audio inputs are not supported. |
| **Output** | 768-dimensional embeddings, truncatable to 512 / 256 / 128 (Matryoshka) |
| **Similarity** | Cosine similarity (L2-normalize embeddings) |
| **Max sequence length used in training** | 512 tokens (base model supports up to 8K) |
| **Precision** | Use FP32 or BF16. The base model card warns against FP16. |
| **Developed by** | Syed Muhammad Hassan (Team Qubit)|
---
## Intended use
**Intended:**
- Semantic and hybrid (dense + BM25) retrieval over NASA combustion, microgravity, and spacecraft fire-safety technical reports.
- First-stage candidate retrieval in a RAG or research-assistant pipeline where results are shown with their sources.
**Not intended:**
- Fire-safety decision-making, material qualification, or any safety-critical judgment. Retrieval relevance is not scientific correctness, and the system must not be used as a substitute for expert review.
- General-purpose or multilingual embedding. Use the base model for that.
- Answer verification. This model ranks passages. It does not check claims.
---
## How to use
### Prefixes are required
The model was trained with fixed text prefixes. **Use exactly these formats for queries and documents**, or retrieval quality will silently degrade.
```python
QUERY_PREFIX = "task: search result | query: "
def fmt_query(q: str) -> str:
return QUERY_PREFIX + " ".join(str(q).split())
def fmt_doc(title: str, text: str) -> str:
t = " ".join((title or "").split()) or "none"
return f"title: {t} | text: {str(text).strip()}"
```
### Sentence Transformers
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("YOUR_HF_USERNAME/flameledger-embeddinggemma2-nasa-combustion")
model.max_seq_length = 512
queries = [fmt_query("At what oxygen concentration does flame spread stop in low-speed opposed flow?")]
docs = [
fmt_doc("Combustion of solid fuel in very low speed oxygen streams",
"Flame spread over thin solid fuel in quiescent and low-speed flows ..."),
fmt_doc("Particle and Smoke Detection on ISS for Next Generation Smoke Detectors",
"Smoke detector response depends on particle size distribution ..."),
]
# Prefixes are added manually above. Use .encode() so no extra prompt is prepended.
q = model.encode(queries, normalize_embeddings=True)
d = model.encode(docs, normalize_embeddings=True)
scores = q @ d.T # cosine similarity
print(scores)
```
### Using a smaller dimension (Matryoshka)
```python
import numpy as np
def truncate(emb, dim=256):
emb = emb[:, :dim]
return emb / np.linalg.norm(emb, axis=1, keepdims=True)
q256, d256 = truncate(q, 256), truncate(d, 256)
```
Re-normalize after truncating. See the [dimension results](#effect-of-embedding-dimension) before going below 256.
### Hybrid retrieval (recommended)
The best configuration in our evaluation fuses dense and lexical rankings with Reciprocal Rank Fusion:
- Dense: cosine similarity at 768 dimensions.
- Lexical: BM25 (k1 = 1.5, b = 0.75) over `title + " " + text`, tokenized as lowercase alphanumerics (keeping internal `.` and `-`, e.g. `6061-t6`).
- Fusion: RRF with k = 60 over the top 100 of each ranking.
A reference implementation (`HybridRetriever`, `BM25`, `rrf_scores`) is in the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger).
---
## Training
### Data
Training data is synthetic. No human-written training questions were used.
| Stage | Count |
|---|---|
| NTRS records fetched | 416 |
| On-topic papers after de-duplication and topic filter | 399 |
| Passages (chunks) | 486 (399 abstracts + 87 body chunks from 10 full-text PDFs) |
| Valid teacher queries after rule-based validation | 1,412 (train 1,128 / dev 134 / test 150) |
| Training triplets after round-trip filter | 1,025 (dev: 134) |
Most papers are represented by their abstract only, because scanned PDFs were skipped rather than guessed.
**Query generation.** Gemma 4 12B instruction-tuned (`google/gemma-4-12B-it`) wrote three queries per passage in three styles: `keyword`, `question`, and `scenario` (an engineering need the passage helps with). Teacher output was kept only if it passed rule checks:
- length limits per style,
- no meta-references such as "the passage" or "this paper",
- no verbatim copying (no 7-word run copied from the passage),
- every multi-digit or decimal number in the query must appear in the passage (blocks invented numbers),
- de-duplication.
**Round-trip filter (train split only).** Queries for which the base model ranked the source passage outside its top 15 were dropped (kept 1,025 of 1,128, or 91%).
**Hard negatives.** Three per query, mined with the base model from the same split only, never from the same paper, skipping near-duplicates (cosine ≥ 0.92) and candidates scoring within 5% of the positive, to reduce false negatives.
**Leak-free split.** Train / dev / test are split **by paper** (321 / 39 / 39 papers), with papers sharing a normalized title kept in the same split.
### Procedure
| | |
|---|---|
| **Loss** | `MatryoshkaLoss` (dims 768, 512, 256, 128) wrapping `CachedMultipleNegativesRankingLoss` |
| **Epochs** | 3 (17 steps per epoch) |
| **Batch size** | 64 (cached MNRL mini-batch 16) |
| **Learning rate** | 2e-5, warmup ratio 0.1, weight decay 0.01 |
| **Precision** | FP32 master weights with BF16 autocast |
| **Batch sampler** | `NO_DUPLICATES` |
| **Checkpoint selection** | Best dev nDCG@10 (evaluated every 8 steps) |
| **Max sequence length** | 512 |
| **Seed** | 42 |
| **Hardware** | 1× NVIDIA L4 (24 GB), Google Colab |
| **Software** | sentence-transformers 6.2.0, transformers 5.22.1, PyTorch 2.11.0 |
Dev-set results before and after fine-tuning (dev was used to pick the checkpoint, so treat these as optimistic):
| Dev metric | Base | Fine-tuned |
|---|---|---|
| nDCG@10 | 0.877 | 0.916 |
| MRR@10 | 0.844 | 0.890 |
| Accuracy@1 | 0.769 | 0.821 |
| Accuracy@5 | 0.948 | 0.993 |
---
## Evaluation
All numbers below are from the training notebook's scorecard. Recall@k is the fraction of queries whose labeled source passage appears in the top k results. Confidence intervals are 95% paired bootstrap.
Full scorecard output from the training notebook: [`scorecard.md`](./scorecard.md).
### Evidence quality
| Query set | n | Origin | Strength |
|---|---|---|---|
| Synthetic test | 150 | Teacher-written, same pipeline and style as training queries | Weakest. Optimistic. |
| Teacher-drafted gold | 23 | Harder, less keyword-matching questions drafted by the teacher. **Not human-reviewed.** | Weak (small n, unreviewed) |
| Human-written gold | 0 | none | Not available |
**No human-written evaluation questions exist for this model.** The numbers measure retrieval on teacher-written queries and should not be read as deployed accuracy.
### Synthetic test, held-out papers only (n = 150)
Candidates are the 51 passages from the 39 test papers. This is the cleanest setting, but it is easy: scores saturate.
| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.800 | 0.967 | 1.000 | 0.874 |
| Base, dense 768 | 0.713 | 0.967 | 0.993 | 0.811 |
| **Fine-tuned, dense 768** | 0.793 | 0.993 | 1.000 | 0.871 |
| Base + BM25 (RRF) | 0.787 | 0.987 | 1.000 | 0.873 |
| **Fine-tuned + BM25 (RRF)** | **0.813** | **1.000** | 1.000 | **0.891** |
### Synthetic test, full archive (n = 150)
Candidates are all 486 passages. This includes passages from training papers that the fine-tuned model saw during training, so it is less clean than the table above and likely favors the fine-tuned model.
| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.647 | 0.900 | 0.940 | 0.752 |
| Base, dense 768 | 0.527 | 0.840 | 0.893 | 0.652 |
| **Fine-tuned, dense 768** | 0.647 | 0.920 | 0.953 | 0.747 |
| Base + BM25 (RRF) | 0.673 | 0.913 | 0.967 | 0.776 |
| **Fine-tuned + BM25 (RRF)** | **0.713** | **0.940** | **0.987** | **0.807** |
### Teacher-drafted gold, unreviewed (n = 23)
| Method | Recall@1 | Recall@5 | Recall@10 | MRR@10 |
|---|---|---|---|---|
| BM25 | 0.304 | 0.609 | 0.739 | 0.436 |
| Base, dense 768 | 0.391 | 0.609 | 0.739 | 0.506 |
| Fine-tuned, dense 768 | 0.478 | 0.609 | 0.696 | 0.537 |
| Base + BM25 (RRF) | 0.261 | 0.565 | 0.696 | 0.422 |
| Fine-tuned + BM25 (RRF) | 0.391 | 0.696 | 0.696 | 0.505 |
### Fine-tuned minus base (95% paired bootstrap CI)
| Query set | Comparison | Recall@5 | MRR@10 |
|---|---|---|---|
| Synthetic, held-out papers | dense 768 | +0.027 [+0.007, +0.060] | +0.060 [+0.031, +0.092] |
| Synthetic, held-out papers | + BM25 (RRF) | +0.013 [+0.000, +0.033] | +0.018 [-0.002, +0.037] |
| Synthetic, full archive | dense 768 | +0.080 [+0.033, +0.127] | +0.095 [+0.053, +0.141] |
| Synthetic, full archive | + BM25 (RRF) | +0.027 [+0.000, +0.060] | +0.031 [-0.001, +0.065] |
| Teacher-drafted gold (n = 23) | dense 768 | +0.000 [-0.130, +0.130] | +0.031 [-0.021, +0.099] |
| Teacher-drafted gold (n = 23) | + BM25 (RRF) | +0.130 [+0.000, +0.261] | +0.083 [+0.014, +0.164] |
### How to read these results
- Fine-tuning gives a **statistically supported improvement to dense retrieval** on synthetic queries (confidence intervals exclude zero for Recall@5 and MRR@10).
- Once BM25 is added, the improvement over base + BM25 is **small**, and several intervals touch or cross zero.
- On the harder drafted-gold set, dense retrieval shows **no detectable difference** (n = 23 is too small to conclude either way).
- **BM25 alone is competitive**: on held-out synthetic queries its MRR@10 (0.874) matches the fine-tuned dense model (0.871). The practical recommendation is hybrid retrieval, not dense alone.
- Synthetic queries share the teacher model's phrasing, so all synthetic scores are optimistic.
### Effect of embedding dimension
Fine-tuned model, synthetic test (n = 150):
| Dimension | Held-out Recall@5 | Held-out MRR@10 | Full-archive Recall@5 | Full-archive MRR@10 |
|---|---|---|---|---|
| 768 | 0.993 | 0.871 | 0.920 | 0.747 |
| 512 | 0.987 | 0.880 | 0.907 | 0.752 |
| 256 | 0.973 | 0.871 | 0.893 | 0.734 |
| 128 | 0.967 | 0.865 | 0.833 | 0.720 |
768 and 512 are effectively equivalent here. 256 is a reasonable storage trade-off. 128 loses noticeable recall on the full archive, and the base model card also advises validating 128-d on your own workload.
---
## Limitations
- **Very small training set.** 1,025 triplets from 486 passages (399 of them abstracts). The model has seen a narrow slice of the literature.
- **Synthetic supervision and evaluation.** Training and test queries come from one teacher model. There is no human-written evaluation set, and the 23-question drafted set was not human-reviewed.
- **Modest, partly uncertain gains.** See [How to read these results](#how-to-read-these-results).
- **Narrow domain.** Trained only on NASA combustion and fire-safety text. Out-of-domain behavior was not evaluated and may be worse than the base model.
- **English only.** Multilingual ability of the base model was not evaluated after fine-tuning.
- **Abstract-heavy corpus.** Most papers contribute only their abstract, so retrieval of detailed in-body conditions (tables, run parameters) is largely untested.
- **Sequence length.** Fine-tuned at 512 tokens. Long-document behavior was not evaluated.
- **Text only.** The vision and audio towers were removed from this export.
- **Retrieval is not correctness.** A relevant passage can still be misread, mismatched to a condition, or superseded. Always show sources and keep a human in the loop.
## Responsible use
This model supports literature search. It is **not** a NASA product, is not endorsed by NASA or Google, and must not be used as a fire-safety decision system.
## AI and tool disclosure
> We fine-tuned Google's EmbeddingGemma 2 to retrieve relevant NASA combustion research. Gemma 4 12B generated synthetic queries for training. FlameLedger combines embedding-based retrieval with BM25 keyword search.
AI-assisted engineering tools (Codex, Antigravity) were used for software development and deployment. Gemini (`gemini-3.1-flash-lite`) is used in the FlameLedger application, not in this model, to draft cited briefings from retrieved evidence.
## License and attribution
- This model is a derivative of `google/embeddinggemma-2`, released by Google DeepMind under Apache 2.0, and is released here under **Apache 2.0**. The upstream license page links Google's Gemma 4 license terms. Please review the [base model card](https://huggingface.co/google/embeddinggemma-2) for the terms that apply to the original weights.
- Training queries were generated with `google/gemma-4-12B-it`.
- The training corpus is derived from public NASA NTRS records. **No corpus text is redistributed with this model.**
## Citation
```bibtex
@misc{flameledger_encoder_2026,
title = {FlameLedger Encoder: EmbeddingGemma 2 fine-tuned for NASA combustion research retrieval},
author = {{Syed Muhammad Hassan}, {Team Qubit}},
year = {2026},
howpublished = {Hugging Face model repository},
note = {NASA Space Apps Challenge 2026, Flame in Freefall. Code: https://github.com/Muhammad-Hassan12/FlameLedger}
}
```
Please also cite the base model, EmbeddingGemma 2 (Google DeepMind).
## Contact
Open an issue on the [FlameLedger repository](https://github.com/Muhammad-Hassan12/FlameLedger).