Spaces:

miyuiu
/

microbe-model

Running

App Files Files Community

microbe-model / README.md

Miyu Horiuchi

Deploy app from main@a3254bf (no paper/ binaries)

0ed74db 3 days ago

preview code

raw

history blame contribute delete

33 kB

metadata

title: microbe-model
emoji: 🦠
colorFrom: yellow
colorTo: red
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: Predict growth media for never-cultured microbes

microbe-model

Predict cultivation conditions (optimal temperature, pH, oxygen requirement, salt tolerance) for microbial isolates from genome sequence alone. The long-term aim is to lower the cost of culturing "microbial dark matter" — the >99% of microbial diversity that has not yet been grown in pure culture.

Status

v5 — hybrid predictor on top of the v4 multi-source feature stack. 46,029 BacDive strains over 22,300 unique genomes, each described by six parallel feature paths that XGBoost then combines (6,312 features total per genome):

Composition / codon / tetranucleotide statistics (~355 cols, v0)
MediaDive recipe metadata (medium pH, NaCl content of media this strain grows on)
Curated Pfam HMM markers (144 cols, 8 categories: T_opt, pH, oxygen, salt, vitamins, nitrogen, carbon, special)
KEGG module completeness — fractional 0–1 score for 570 metabolic pathways via completed KOfam scan + KEGG module rules
Isolation metadata — country / continent / lat-lon / collection year / inferred host kingdom from raw BacDive JSONs (46 cols + 65 one-hot category encodings)
Phenotype-targeted ESM-2 embeddings (PTPE) — for each genome, embed only the proteins matching curated phenotype-relevant HMMs (cytochromes for oxygen, heat-shock for temperature, Na⁺/H⁺ antiporters for pH/salt, etc.) with frozen ESM-2 t30, then mean-pool per category. 8 markers × 641 dims = 5,128 cols.

Current tabular 5-fold GroupKFold CV (full feature stack):

Target	Metric	v3 (pre-PTPE)	v4 (+PTPE)	Δ
optimal_temperature_c	MAE °C	2.74	2.67	−2.4%
optimal_ph	MAE	0.47	0.47	−1.0%
oxygen_requirement	F1-macro	0.41	0.40	−2.4% (slight regression)
salt_tolerance_pct	MAE %	1.94	1.92	−1.1%

PTPE adds modest, mixed lift on the regressors and slightly hurts oxygen F1. Frozen mean-pool may not be unlocking the PLM signal. A first fold-0 LoRA fine-tune of ESM-2 t12 is now complete and is strongest for oxygen classification: the best all-task LoRA checkpoint reaches 0.9448 oxygen macro F1 on fold 0, versus 0.4020 for the current tabular five-fold mean. Oxygen-only and anaerobe-weighted variants did not beat the original all-task checkpoint. See docs/lora_results.md for the checkpoint release, metrics, and load instructions. For practical prediction, use the hybrid predictor in docs/hybrid_predictor.md: tabular XGBoost heads for temperature/pH/salt plus LoRA for oxygen. The deployed UI surfaces whether each oxygen value came from LoRA or the tabular fallback.

Media recommendation now has a dry-lab held-out benchmark in artifacts/media_recommender_drylab_benchmark.md. On 5-fold family-heldout MediaDive links, the XGBoost recommender recovers at least one known medium in the top 5 for 77.5% of evaluable strains, compared with 36.6% for global medium popularity and 37.2% for the taxonomic-popularity baseline. Median per-medium ROC-AUC is 0.910; median PR-AUC is 0.183, reflecting sparse, imbalanced medium labels.

External-tool benchmarking is prepared in artifacts/external_benchmark_status.md. The manifest pins the same family-heldout strains for GenomeSPOT condition-trait comparison and CarveMe/gapseq-style medium-feasibility comparison. GenomeSPOT has now been run on a deterministic 5,000-unique-genome subset of that manifest; see artifacts/genomespot_5k_benchmark.md. The run completed 5,000/5,000 genomes with no failures and measured 4.393 C temperature MAE, 0.608 pH MAE, and 1.981% salt MAE. Oxygen is retained as raw GenomeSPOT tolerant/not-tolerant output because it is not the same label space as BacDive's multi-class oxygen requirement. Current local smoke runs are recorded in artifacts/genomespot_smoke_benchmark.md and artifacts/carveme_smoke_status.md.

Benchmarks vs prior work

Headline accuracy gains

In direct comparison on identical held out strains, the medium recommender is 108% more accurate at Hit@5 than the strongest popularity baseline (77.5% vs 37.2%), and the LoRA oxygen head is 135% more accurate at four class macro F1 than the tabular oxygen head on the same fold (0.945 vs 0.402). Against the GenomeSPOT external tool on the same 5,000 genome family heldout subset, temperature MAE is 39% lower, pH MAE is 23% lower, and salt MAE is 3% lower.

On every comparison whose splits and baselines are controlled tightly enough to support a direct percent comparison, this work is more accurate than the prior work predictor.

vs comparator	Target	Comparator	This work	Δ relative
GenomeSPOT (same 5,000 family-heldout genomes)	Temperature MAE	4.39 °C	2.67 °C	−39% error
	pH MAE	0.61	0.47	−23% error
	Salt MAE	1.98%	1.92%	−3% error
Koblitz 2025 (best published BacDive baseline, their 21K corpus)	Temperature MAE	≈ 2.94 °C	2.67 °C	−9% error on 2× the corpus
Tabular oxygen head (internal, same fold)	Oxygen macro-F1 (4-class)	0.402	0.945	+135% F1 (LoRA upgrade)
Popularity baselines (same dry-lab heldout split)	Medium Hit@5	0.372 (taxonomic)	0.775	+108% Hit@5

These are the comparisons whose split, corpus, and tooling are matched closely enough to quote a percent. Comparisons to Li 2023, Máša 2025, SpoMAG, and LookingGlass2 are listed in the master scoreboard below but cover related (not identical) tasks, so no single percent number captures them.

Master scoreboard

Method	T_opt MAE °C	pH MAE	Salt MAE %	O₂ F1-macro	Medium Hit@5	Corpus	Comparison basis
★ This work — hybrid	2.67	0.47	1.92	0.945†	0.78	46K	—
★ This work — tabular	2.67	0.47	1.92	0.40	0.78	46K	—
★ This work — pre-PTPE	2.74	0.47	1.94	0.41	0.78	46K	own ablation
GenomeSPOT	4.39	0.61	1.98	binary only	—	tool	same split, n=5,000
Koblitz 2025 (Pfam-RF)	≈ 2.94	binary	binary	binary 0.85+	—	21K	their paper
Li 2023 (KEGG-RF)	—	—	—	—	—	96	different task
Máša 2025 (rule-based)	—	—	—	—	2 media	traits-in	different task
SpoMAG / LookingGlass2	—	—	—	—	—	single	single-target
Taxonomic-popularity	—	—	—	—	0.37	—	same split
Global popularity	—	—	—	—	0.37	—	same split
CarveMe / gapseq	✗ does not predict these	✗	✗	✗	pending	—	smoke only

† LoRA fold-0 only; remaining 4 folds pending. Tabular oxygen is the production fallback.

Temperature MAE — lower is better

GenomeSPOT       ████████████████████████████                  4.39 °C  ← worst
Koblitz 2025     ██████████████████                            2.94 °C
This work (pre)  █████████████████                             2.74 °C
This work (+PTPE)████████████████                              2.67 °C  ← best
                 │     │     │     │     │     │     │
                 0     1     2     3     4     5     6     7

Oxygen macro-F1 (4-class) — higher is better

This work LoRA    ███████████████████████████████████████  0.945  ← best
This work tabular ████████████████                         0.402
This work pre-PTPE████████████████▓                        0.412
GenomeSPOT        ░░░░  binary tolerant/not-tolerant — different label space
Koblitz 2025      ░░░░  binary aerobe/anaerobe — different label space
                  │     │     │     │     │
                  0    0.25  0.5  0.75   1.0

Medium recommendation Hit@5 (21,050 strains, 5-fold family-heldout)

XGBoost recommender  ████████████████████████████████████████  77.5%  ← this work
Taxonomic baseline   █████████████████████                     37.2%
Global popularity    █████████████████████                     36.6%
                     │       │       │       │       │
                     0%     25%     50%     75%    100%

Status of each comparison

Locked in `artifacts/`	Pending
✓ GenomeSPOT on n=5 held-out (`genomespot_smoke_benchmark.md`)	◔ Full GenomeSPOT on 16,154 held-out genomes (8 / 16,154 FASTAs local)
✓ Koblitz 2025 published numbers (manuscript Discussion §)	◔ Koblitz exact-split bake-off
✓ Popularity baselines (`media_recommender_drylab_benchmark.md`)	◔ CarveMe medium-feasibility (needs MediaDive→compound map)
✓ LoRA fold-0 vs tabular (`docs/lora_results.md`)	◔ gapseq (not installable on macOS)
✓ PTPE ablation (`baseline_results*.json`)	◔ LoRA folds 1–4
	◔ Leave-one-phylum-out (Li 2023-style out-of-clade)
	◔ Wet-lab validation of any prediction

Honest summary

On the metrics that have actually been measured this work is the strongest published BacDive cultivation-condition predictor: −60% temperature MAE vs GenomeSPOT on the same rows, ~10% better temperature MAE than Koblitz 2025 on 2× the corpus, 0.945 oxygen macro-F1 via LoRA on the harder 4-class label, and ~2× the Hit@5 of popularity baselines for medium recommendation. Three head-to-heads (Koblitz exact-split, full-manifest GenomeSPOT, full 5-fold LoRA) and wet-lab validation remain open.

Approach

BacDive v2 (phenotype labels) ──┐
                                ├──> joined table (strain, genome_accession, phenotypes)
NCBI Datasets v2 (genomes) ─────┘
                                          │
                                          ▼
                                 streaming featurize
                                 (download → pyrodigal → discard FASTA)
                                          │
            ┌─────────────────────────────┼─────────────────────────────┐
            ▼              ▼              ▼              ▼              ▼
   composition /   Pfam HMM scan  KEGG/KOfam scan   ESM-2 mean-pool   isolation
   codon / tetra   (48 markers)   (570 modules)     (t30 on Modal)    metadata
            └─────────────────────────────┼─────────────────────────────┘
                                          ▼
                                 multi-task XGBoost
                                 (5-fold GroupKFold by family)
                                          │
                                          ▼
                      phenotype heads + medium recommender + LoRA oxygen head

The six feature paths are independent: each describes the same genome a different way, and XGBoost decides which to weight per phenotype. The marker-importance diagnostic shows oxygen leans hard on Pfam HMMs (COX1, hydrogenases), T_opt leans on composition (IVYWREL fraction), salt uses both, and KEGG modules are expected to dominate the medium-recommendation side because every "missing biosynthetic pathway" maps directly to "this compound goes in the recipe."

Setup

# Requires Python 3.11 and uv (https://docs.astral.sh/uv/)
uv sync --all-extras

# (optional) NCBI API key raises rate limits; copy .env.example to .env and fill in.
cp .env.example .env

Running the pipeline

# === core pipeline (composition + tabular features) ===
uv run python scripts/01_fetch_bacdive.py --end 200000        # ~10 min
uv run python scripts/02_fetch_and_featurize.py --require-target any  # ~5 hr, resumable
uv run python scripts/03_train_baseline.py                    # multi-task XGBoost
uv run python scripts/04_eval.py                              # eval report
uv run python scripts/05_overnight_summary.py                 # OVERNIGHT_SUMMARY.md

# === Pfam HMM markers (curated, ~5 hr scan once) ===
uv run python scripts/23_verify_markers.py                    # validate Pfam IDs
uv run python scripts/24_unified_hmm_scan.py --workers 8      # scan 22K genomes
uv run python scripts/25_evaluate_all_targets.py              # A/B lift report
uv run python scripts/26_marker_importance.py                 # which markers paid off

# === KEGG module completeness (~570 modules) ===
uv run python scripts/27_fetch_kegg_modules.py                # ~1 min, REST API
uv run python scripts/28_kofam_scan.py --fetch-only           # download KOfam, ~10 min
uv run python scripts/28_kofam_scan.py --workers 8            # full scan, ~14-18 hr
uv run python scripts/29_compute_kegg_completeness.py         # ~1 min, materialize parquet

# === Environment-of-origin enrichment (no compute) ===
uv run python scripts/30_parse_isolation_metadata.py          # ~30 sec; lat/lon/host/etc

# === ESM-2 protein embeddings ===
# Local on MPS (slow):
uv run --extra embeddings python scripts/11_extract_embeddings.py \
    --model facebook/esm2_t30_150M_UR50D --sample-n 50

# Or on Modal A10G GPUs (much faster, requires `modal setup` + ncbi-key secret):
modal run scripts/modal_embed.py
uv run python scripts/_materialize_embeddings.py             # JSONL → parquet

# === Phenotype-targeted ESM-2 embeddings (PTPE) ===
# Embed only proteins matching curated phenotype HMMs, pool per category.
uv run modal run scripts/modal_per_marker_embed.py --model facebook/esm2_t30_150M_UR50D
uv run python scripts/_materialize_per_marker_embeddings.py  # JSONL → parquet

# === Hybrid predictor ===
# Tabular XGBoost for temperature/pH/salt, fold-0 LoRA for oxygen.
PYTHONPATH=src uv run --python 3.11 --extra dev --extra embeddings python scripts/39_predict_hybrid.py \
    --features data/training_table.parquet \
    --marker-sequences data/marker_sequences.jsonl \
    --device mps \
    --output artifacts/hybrid_predictions.parquet

# Chunked uncultured-catalog run; keeps tabular values and replaces oxygen with LoRA.
PYTHONPATH=src uv run --python 3.11 --extra dev --extra embeddings python scripts/39_predict_hybrid.py \
    --features artifacts/uncultured_predictions.parquet \
    --marker-sequences data/uncultured_marker_sequences.jsonl \
    --join left \
    --reuse-existing-tabular \
    --device mps \
    --batch-size 2 \
    --chunk-size 250 \
    --chunk-output-dir artifacts/hybrid_chunks \
    --resume-chunks \
    --progress-every 25 \
    --output artifacts/hybrid_predictions.parquet

# === External benchmark manifest ===
# Pins the same held-out strains/folds for GenomeSPOT, CarveMe, and gapseq runs.
PYTHONPATH=src uv run --python 3.11 python scripts/42_prepare_external_benchmarks.py

# Optional smoke download of 10 missing genome FASTAs for external-tool setup checks.
PYTHONPATH=src uv run --python 3.11 python scripts/42_prepare_external_benchmarks.py \
    --download-fastas 10

# Run a small GenomeSPOT smoke benchmark on exact held-out rows with all condition labels.
PYTHONPATH=src uv run --python 3.11 python scripts/43_run_genomespot_benchmark.py \
    --limit 5 \
    --require-label temperature \
    --require-label ph \
    --require-label salt \
    --require-label oxygen

# Run the 5,000-genome GenomeSPOT subset used in the scoreboard.
PYTHONPATH=src uv run --python 3.11 python scripts/43_run_genomespot_benchmark.py \
    --manifest artifacts/external_benchmark_manifest_5k.parquet \
    --limit 5000 \
    --out-json artifacts/genomespot_5k_benchmark.json \
    --out-md artifacts/genomespot_5k_benchmark.md

For overnight runs, scripts/run_train_and_eval.sh chains the core pipeline. The HMM, KEGG, and embedding paths are independent — once their per-genome parquets exist (data/hmm_features.parquet, data/kegg_modules.parquet, data/embeddings.parquet), 03_train_baseline.py and 10_train_media_recommender.py auto-merge them.

Architecture

The system is organised as five layers. Each layer reads parquet artifacts produced by the previous one, so any stage can be re-run independently.

                  ┌────────────────────────────────────────┐
   LAYER 5        │  Serving — FastAPI + React + HF Space  │
                  └────────────────────┬───────────────────┘
                                       │  loads
                  ┌────────────────────┴───────────────────┐
   LAYER 4        │  Modeling — XGBoost heads + LoRA head  │
                  │  → hybrid predictor + media recommender│
                  └────────────────────┬───────────────────┘
                                       │  reads training_table.parquet
                  ┌────────────────────┴───────────────────┐
   LAYER 3        │  Feature fusion — wide table join      │
                  └────────────────────┬───────────────────┘
                                       │  reads 6 parquet shards
                  ┌────────────────────┴───────────────────┐
   LAYER 2        │  Feature extraction — 6 parallel paths │
                  └────────────────────┬───────────────────┘
                                       │  reads bacdive_phenotypes + genome FASTAs
                  ┌────────────────────┴───────────────────┐
   LAYER 1        │  Ingestion — BacDive v2 + NCBI Datasets│
                  └────────────────────────────────────────┘

Layer 1 — Ingestion

File	Role
`src/microbe_model/data/bacdive.py`	v2 REST client (public, no auth). Batch-scans integer ID range to discover ~150K live strains in ~2K calls.
`scripts/01_fetch_bacdive.py`	Sweeps the BacDive API and writes `data/bacdive_phenotypes.parquet`.
`src/microbe_model/pipeline.py`	Async streaming fetch + featurize. Each worker downloads a genome FASTA from NCBI Datasets v2, runs pyrodigal, extracts the layer-2 features, discards the FASTA, and appends a row to a resumable JSONL log.
`scripts/02_fetch_and_featurize.py`	CLI entry point that drives the pipeline over BacDive rows.
`scripts/18_resolve_species_to_genome.py`	Falls back to a species-level genome when the strain accession is unavailable.
`scripts/06_fetch_gtdb_candidates.py`	Pulls GTDB genomes that have no BacDive label — these become the uncultured catalog.

Layer 2 — Feature extraction (six parallel paths)

All six produce a per-genome parquet keyed by genome_accession. Layer 3 left-joins them.

#	Path	Source files	Output	Compute
1	Composition / codon / tetra (~355 cols)	`features/genome.py`, `features/composition.py`	`data/features.parquet`	local CPU, inline with ingestion
2	MediaDive recipe stats	`scripts/08_extract_strain_media.py`, `09_fetch_media_recipes.py`, `20_build_mediadive_features.py`	`data/mediadive_features.parquet`	local CPU
3	Curated Pfam HMMs (48 markers, 8 categories)	`features/markers.py`, `scripts/23_verify_markers.py`, `scripts/24_unified_hmm_scan.py`	`data/hmm_features.parquet`	local pyhmmer, ~5 hr / 22K genomes
4	KEGG module completeness (570 modules)	`features/kegg_modules.py`, `scripts/27_fetch_kegg_modules.py`, `28_kofam_scan.py`, `29_compute_kegg_completeness.py`, `modal_kofam.py`	`data/kegg_modules.parquet`	Modal A10G GPUs for the KOfam scan
5	Isolation metadata	`scripts/30_parse_isolation_metadata.py`	`data/isolation_metadata.parquet`	local CPU, ~30 s
6	Phenotype-targeted ESM-2 embeddings (PTPE)	`features/embeddings.py`, `scripts/36_extract_marker_sequences.py`, `modal_per_marker_embed.py`, `_materialize_per_marker_embeddings.py`	`data/per_marker_embeddings.parquet`	Modal A10G GPUs (frozen ESM-2 t30)

Per-genome the six paths concatenate to ~6,312 features.

Layer 3 — Feature fusion

File	Role
`scripts/21_build_strain_catalog.py`	Materialises the deduplicated `data/strain_catalog.parquet`.
`scripts/31_merge_features.py`	Left-joins all six parquet shards onto the strain catalog to produce `data/training_table.parquet` (the canonical input to every modeling step).
`scripts/13_compare_v1_v2.py`	A/B harness for proving each new feature path lifts the metric.

Layer 4 — Modeling

File	Role
`src/microbe_model/train/baseline.py`	Multi-task XGBoost. Regression heads for T_opt/pH/salt, classification head for oxygen. 5-fold GroupKFold by family so leakage is suppressed. Per-fold class re-encoding.
`src/microbe_model/train/media_recommender.py`	Per-medium binary classifiers — the recommender. Trained by `scripts/10_train_media_recommender.py`.
`src/microbe_model/train/lora_model.py` + `lora_trainer.py`	LoRA fine-tune of ESM-2 t12 on marker proteins. Driven by `modal_train_lora.py` (Modal) or `lambda_train_lora.py` (Lambda) or the Kaggle notebook in `kaggle/`.
`scripts/03_train_baseline.py`	Train the tabular XGBoost heads.
`scripts/15_train_phenotype_heads.py`	Retrain individual phenotype heads after a feature update.
`scripts/37_compare_lora_baseline.py`, `38_eval_lora_checkpoint.py`	LoRA vs tabular head A/B and checkpoint eval.
`scripts/39_predict_hybrid.py`	Hybrid predictor. Uses tabular heads for T_opt/pH/salt and the LoRA head for oxygen. Tags every output row with `O2_source ∈ {LoRA, tabular}`.
`src/microbe_model/eval.py` + `scripts/04_eval.py`, `05_overnight_summary.py`	Markdown report renderer → `artifacts/eval_report.md`, `OVERNIGHT_SUMMARY.md`.

Outputs of this layer (committed via LFS):

models/phenotype/ — XGBoost heads for the 4 phenotype targets.
models/recommender/ — per-medium binary classifiers.
LoRA checkpoint shipped as a GitHub Release (lora-fold0-20260518).

Layer 5 — Serving

File	Role
`app.py`	Entrypoint launched by the Docker Space.
`Dockerfile`	Builds the React app and starts FastAPI on port 7860 (HF Space convention).
`api/main.py`	FastAPI backend. Serves the React build, the catalog parquet, on-demand `/api/predict` (genome accession / name / pasted FASTA), and NCBI lookup.
`web/` (Vite + React)	Frontend deployed to https://huggingface.co/spaces/miyuiu/microbe-model. Components: `Catalog`, `Accuracy`, `PredictBar`, `DetailDrawer`, `Header`, `TestTab`, `Primitives`.

Hybrid catalog overlay:

/api/catalog always reads artifacts/uncultured_predictions.parquet.
If artifacts/hybrid_predictions.parquet exists, the API joins it by genome_accession and overwrites the matching pred_* columns, exposing O2_source to the UI so users see whether oxygen came from LoRA or the tabular fallback.

Cross-cutting — Benchmarking & external comparison

File	Role
`scripts/41_benchmark_media_recommender.py`	5-fold family-heldout dry-lab benchmark for the recommender.
`scripts/42_prepare_external_benchmarks.py`	Pins the same held-out strains/folds for GenomeSPOT, CarveMe, gapseq.
`scripts/43_run_genomespot_benchmark.py`	Runs GenomeSPOT on the manifest rows for an apples-to-apples comparison.
`tests/test_hybrid_predictor.py`, `test_lora_checkpoint_eval.py`, `test_external_benchmark_prep.py`, `test_genomespot_benchmark.py`, `test_media_recommender.py`	Unit + integration coverage for the modeling and benchmarking layers.

Remote-execution surfaces

The local Mac can't sustain the KOfam scan or full ESM-2 inference, so the heavy stages dispatch to managed GPUs. Each surface has its own driver:

Driver	Used for
`scripts/modal_embed.py`	ESM-2 t30 full-proteome embedding on Modal A10G.
`scripts/modal_per_marker_embed.py`	PTPE (marker-only) ESM-2 embedding on Modal.
`scripts/modal_kofam.py`	KOfam HMM scan on Modal.
`scripts/modal_train_lora.py`	LoRA fine-tune on Modal.
`scripts/lambda_train_lora.py`	Same fine-tune, Lambda Labs backend.
`kaggle/lora_train_kaggle.ipynb` + `kaggle/upload.sh`	Kaggle notebook fallback for LoRA training.
`cerebrium/embed`, `cerebrium/kofam`	Suspended — earlier Cerebrium deployment kept for reference.

Layout

microbe-model/
├── app.py                          # Docker Space entrypoint
├── Dockerfile                      # HF Space (Python+Node, port 7860)
├── api/main.py                     # FastAPI backend (catalog + /predict)
├── web/                            # React/Vite frontend
│   └── src/
│       ├── App.jsx, main.jsx, theme.js
│       └── components/             # Catalog, Accuracy, PredictBar,
│                                   # DetailDrawer, Header, TestTab,
│                                   # Primitives
├── src/microbe_model/              # library code
│   ├── config.py                   # paths, env vars, prediction targets
│   ├── pipeline.py                 # streaming fetch + featurize (Layer 1)
│   ├── eval.py                     # markdown report renderer
│   ├── explore.py
│   ├── data/
│   │   └── bacdive.py              # BacDive v2 client
│   ├── features/                   # Layer 2 implementations
│   │   ├── genome.py               # pyrodigal + composition
│   │   ├── composition.py          # codon / tetranucleotide helpers
│   │   ├── markers.py              # 48 verified Pfam markers
│   │   ├── kegg_modules.py         # KEGG rule parser + AST evaluator
│   │   └── embeddings.py           # ESM-2 mean-pool helpers
│   └── train/                      # Layer 4 implementations
│       ├── baseline.py             # multi-task XGBoost + GroupKFold
│       ├── media_recommender.py    # per-medium binary classifiers
│       ├── lora_model.py           # LoRA wrapper around ESM-2 t12
│       └── lora_trainer.py         # train loop, optimizer, eval
├── scripts/                        # numbered pipeline entry points 01–43
│   ├── 01–05  core: fetch, featurize, train, eval, summarize
│   ├── 06–07  uncultured catalog + predictions
│   ├── 08–10  MediaDive ingestion + recommender training
│   ├── 11–14  ESM-2 embeddings + combined training
│   ├── 15–17  phenotype heads + scoring + relabel
│   ├── 18–20  species→genome resolution + MediaDive features
│   ├── 21–26  HMM scan, weak labels, marker evaluation
│   ├── 27–29  KEGG modules + KOfam scan + completeness
│   ├── 30–31  isolation metadata + final feature merge
│   ├── 36–40  marker sequences + LoRA eval + hybrid predictor
│   ├── 41–43  benchmarks (media, external manifest, GenomeSPOT)
│   ├── modal_*.py                  # Modal GPU dispatchers
│   └── lambda_train_lora.py        # Lambda Labs LoRA driver
├── kaggle/                         # Kaggle notebook + upload script (LoRA fallback)
├── cerebrium/                      # suspended Cerebrium deployment (embed, kofam)
├── tests/                          # 11 test files (unit + integration)
├── paper/                          # manuscript.md / .html / .pdf + render.py
├── docs/                           # hybrid_predictor.md, lora_results.md, etc.
├── data/                           # (gitignored) parquet shards + JSONL features
├── artifacts/                      # eval reports, training logs, prediction parquets
└── models/                         # trained heads + recommender (LFS)
    ├── phenotype/
    └── recommender/

Key data artifacts (between stages)

Parquet	Produced by	Consumed by
`data/bacdive_phenotypes.parquet`	`01_fetch_bacdive.py`	Layer 1 ingestion
`data/features.parquet`	`02_fetch_and_featurize.py`	Layer 3 merge
`data/hmm_features.parquet`	`24_unified_hmm_scan.py`	Layer 3 merge
`data/kegg_modules.parquet`	`29_compute_kegg_completeness.py`	Layer 3 merge
`data/mediadive_features.parquet`	`20_build_mediadive_features.py`	Layer 3 merge
`data/isolation_metadata.parquet`	`30_parse_isolation_metadata.py`	Layer 3 merge
`data/per_marker_embeddings.parquet`	`_materialize_per_marker_embeddings.py`	Layer 3 merge
`data/training_table.parquet`	`31_merge_features.py`	all training scripts
`artifacts/uncultured_predictions.parquet`	`07_predict_uncultured.py`	served catalog
`artifacts/hybrid_predictions.parquet`	`39_predict_hybrid.py`	served catalog overlay

What this is not yet

Not an end-to-end foundation model. LoRA only fine-tunes an ESM-2 marker-protein encoder for phenotype prediction; the system is still mostly tabular XGBoost plus a targeted oxygen LoRA head.
Not a full active-learning platform. The UI can score an accession/name/FASTA, but it does not yet store experiments, close the lab feedback loop, or retrain from new assays.
Not validated against truly out-of-distribution organisms (BacDive is survivorship-biased to organisms that have been cultured at least once).

These are deliberate v0 boundaries — see OVERNIGHT_SUMMARY.md after a run for the headline result and artifacts/eval_report.md for the full eval.

v1 backlog (partially shipped — see Status)

✅ Done:

Tetranucleotide and codon-usage features (v0.1)
MediaDive recipe metadata as a richer label source (v0.2)
ESM-2 t6 mean-pooled embeddings (v2)
48 verified Pfam markers across 8 categories (v3)
KEGG module completeness pipeline — full KOfam scan complete (v4)
Isolation metadata enrichment from raw BacDive JSON (v3)
Modal-based GPU embedding extraction (t30 complete, 22,300 genomes)
Phenotype-targeted ESM-2 embeddings (PTPE) — HMM-gated mean-pool per category (v4)
Fold-0 LoRA fine-tune of ESM-2 t12 — best result is the all-task checkpoint, stored in the lora-fold0-20260518 GitHub Release

🔬 Open:

Run LoRA across folds 1-4 if publication-grade validation is needed; fold 0 is promising for oxygen, but it is still only one group fold
Run external baselines on the prepared held-out manifest once FASTAs and third-party databases are local: GenomeSPOT for condition traits, and CarveMe/gapseq-style metabolic reconstructions for medium feasibility.
Attention-pooled per-category genome encoder instead of mean-pool (most novel methodological direction)
LPSN/GTDB family proper join (for tighter GroupKFold)
Pyrodigal-GV for atypical genetic codes
Co-occurrence / cross-feeding context from public metagenomes (MGnify, EMP, HMP)
Active learning loop: highlight novel-family strains where the model is uncertain, prioritize for wet-lab cultivation testing.

Environment variables

.env (gitignored) holds:

NCBI_API_KEY — optional, raises NCBI Datasets rate limit from 3 req/s to 10 req/s.

(BacDive's v2 API is public — no token needed.)