Dimitris commited on
Commit Β·
df889fb
1
Parent(s): ba4739e
docs: add revised plan grounded in current repo state
Browse filesMaps what's done (extraction MVP + UI) to the remaining work and prizes:
interpretation + cross-marker reasoning, offline llama.cpp transition,
fine-tune on Modal, agent trace, eval, and submission artifacts.
PLAN.md
ADDED
|
@@ -0,0 +1,111 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Blood Test Explainer ("Pulse") β Plan, revised against current repo
|
| 2 |
+
|
| 3 |
+
> Grounded in the actual code as of 2026-06-09. Two developers, parallel.
|
| 4 |
+
> Target prizes: **OpenAI + OpenBMB + Modal** + all badges.
|
| 5 |
+
> β οΈ Confirm the real submission deadline first β teams are already final-submitting.
|
| 6 |
+
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## 1. Where we are (DONE β solid foundation)
|
| 10 |
+
|
| 11 |
+
- **Polished Gradio app** (`app.py`): light clinical theme, animated "formation" hero,
|
| 12 |
+
loading/empty states, report cards with status pills, tabs (Report / Values / JSON / Raw),
|
| 13 |
+
responsive. This is genuinely good and is a real head start on the "premium document" wow.
|
| 14 |
+
- **Extraction works** (`src/openbmb_client.py`, `src/document_processing.py`): OpenBMB
|
| 15 |
+
**MiniCPM-V-4.6** (vision) via OpenAI-compatible API. PDFβimages (PyMuPDF), images, and
|
| 16 |
+
text files supported; base64 image payloads; robust JSON-repair parsing.
|
| 17 |
+
- **Schema:** `{marker, value, unit, reference_range, status, source_text, confidence}` + notes.
|
| 18 |
+
- **Env/secret handling** ready (`.env` local, Space secret). Codex is contributing.
|
| 19 |
+
- Extraction prompt is deliberately **extraction-only** ("do not diagnose/interpret").
|
| 20 |
+
|
| 21 |
+
## 2. Blunt gap analysis β what's missing and which prize it unblocks
|
| 22 |
+
|
| 23 |
+
The current app is a beautiful **extractor that calls an external API**. As-is it does not win.
|
| 24 |
+
|
| 25 |
+
| Missing piece | Why it matters | Unblocks |
|
| 26 |
+
|---|---|---|
|
| 27 |
+
| **Interpretation + cross-marker reasoning** | Right now MiniCPM looks like OCR/plumbing. The model must *reason* (e.g. "ALT+AST+GGT all high β liver-enzyme pattern") to be visibly central. | **OpenBMB** "central model" + req #5 |
|
| 28 |
+
| **Cited knowledge base (~40 markers)** | Reliable reference ranges + "what it measures" + "questions for your doctor", grounded not hallucinated. | medical credibility |
|
| 29 |
+
| **Run the model LOCALLY (llama.cpp)** | The current external API call **fails off-grid / "fully offline."** | **off-grid badge** + offline requirement |
|
| 30 |
+
| **Fine-tune + GGUF on Modal** | No fine-tune yet. Needed for the badge + the OpenBMB before/after story. | **fine-tune + quantization badges, Modal prize** |
|
| 31 |
+
| **Agent trace** | Single API call now; needs a visible multi-step pipeline. | req #5 |
|
| 32 |
+
| **Eval harness + before/after** | No metrics yet. | **OpenBMB** proof |
|
| 33 |
+
| **Traces dataset + model card + multi-repo** | Top teams (compliment-forest) do this. | competitiveness |
|
| 34 |
+
| **Video (local run) + social + README lines** | Required submission artifacts. | **general pool** |
|
| 35 |
+
|
| 36 |
+
## 3. The biggest technical fork: getting OFF the external API
|
| 37 |
+
|
| 38 |
+
Off-grid + "fully offline" require the model to **run inside the Space**, no external calls.
|
| 39 |
+
The README already flags this ("API-backed extractor is temporary"). Options:
|
| 40 |
+
|
| 41 |
+
- **Recommended (hybrid):** keep MiniCPM-V for extraction but run it **locally in the Space**
|
| 42 |
+
(MiniCPM-V GGUF via llama.cpp multimodal, or via transformers on **ZeroGPU**), and **fine-tune
|
| 43 |
+
a small MiniCPM *text* model** for the **interpretation + cross-marker reasoning** layer (easier
|
| 44 |
+
to fine-tune, this is the "model is the star" part, runs offline via llama.cpp). Earns off-grid
|
| 45 |
+
+ fine-tune + quantization together.
|
| 46 |
+
- **Simpler fallback:** drop vision; do **local OCR/text extraction** (PyMuPDF text + tesseract)
|
| 47 |
+
β one fine-tuned MiniCPM text model does structuring **and** interpretation. Fully offline,
|
| 48 |
+
easiest fine-tune; weaker on scans/photos.
|
| 49 |
+
- **Decide in Phase B.** Main technical risk = running MiniCPM-V locally; the fallback de-risks it.
|
| 50 |
+
|
| 51 |
+
## 4. Parallel plan from here (2 devs, shared stack, contract-first)
|
| 52 |
+
|
| 53 |
+
**Shared contract to agree first (extend the existing dataclass):** add `Interpretation`
|
| 54 |
+
(`marker, plain, meaning, questions[], citation`) and a `cross_marker: list[str]` + `summary`
|
| 55 |
+
to the result object. Both tracks build against it; Track B can use a fixture until Track A ships.
|
| 56 |
+
|
| 57 |
+
**Module ownership (avoid collisions):**
|
| 58 |
+
- **Track A (model/data):** `src/openbmb_client.py` (+ interpretation), new `src/kb/`,
|
| 59 |
+
`src/reasoning/`, `train/`, `eval/`. Owns: extraction-offline, fine-tune, KB, reasoning, eval.
|
| 60 |
+
- **Track B (product/UI):** `app.py`, report rendering, **interpretation cards**, **cross-marker
|
| 61 |
+
insight section**, **doctor-questions**, **agent-trace panel**, `.html` download, sample report, deploy.
|
| 62 |
+
- **Co-owned:** the result dataclass (the contract).
|
| 63 |
+
|
| 64 |
+
### Phase A β DONE β (extraction MVP + UI shell)
|
| 65 |
+
|
| 66 |
+
### Phase B β Interpretation + KB + reasoning (the win-maker)
|
| 67 |
+
- **Track A:** build the **cited KB** (~40 markers: CBC, metabolic, lipid, thyroid, key vitamins);
|
| 68 |
+
add an **interpretation pass** (grounded in KB + the user's value) and the **cross-marker
|
| 69 |
+
reasoning** (W1). Decide the offline path (Β§3).
|
| 70 |
+
- **Track B:** extend the report to render interpretation per marker + a **cross-marker insights**
|
| 71 |
+
block + **"questions for your doctor"**; add the **agent-trace** panel (Ingest β extract β
|
| 72 |
+
normalize β KB lookup β reason β render) streaming via generator `yield`.
|
| 73 |
+
- **Milestone:** real report β extracted + explained + reasoned document.
|
| 74 |
+
|
| 75 |
+
### Phase C β Offline + fine-tune (the badges)
|
| 76 |
+
- **Track A:** synthetic data (Claude) for extraction/interpretation; **LoRA fine-tune on Modal**
|
| 77 |
+
β merge β **GGUF Q4_K_M**; wire local **llama.cpp** inference; **before/after chart**.
|
| 78 |
+
- **Track B:** deploy to the **org Space** with the local model; verify **zero external calls**;
|
| 79 |
+
graceful failure; downloadable `.html`; bundle a **sample report**.
|
| 80 |
+
- **Milestone:** runs fully offline on a stranger's report; fine-tune metrics in hand.
|
| 81 |
+
|
| 82 |
+
### Phase D β Robustness + artifacts
|
| 83 |
+
- International units (mg/dLβmmol/L), messy formats, unknown markers.
|
| 84 |
+
- Publish **agent traces as a HF dataset** + **model card with GGUF/eval metrics**; multi-repo.
|
| 85 |
+
|
| 86 |
+
### Phase E β Submission
|
| 87 |
+
- Record **2-min video locally** (so "nothing left my laptop" is literally true); **social post**;
|
| 88 |
+
README with every eligibility line; flip repo public if needed; final eval numbers. Freeze + submit.
|
| 89 |
+
|
| 90 |
+
## 5. The two weaknesses we engineer around
|
| 91 |
+
- **W1 β model must be the star:** cross-marker reasoning is a first-class component, not an add-on.
|
| 92 |
+
- **W2 β medical credibility:** every fact from a **cited KB**; "questions for your doctor," never
|
| 93 |
+
diagnosis (the current prompt's restraint is the right instinct β keep it).
|
| 94 |
+
- Privacy is only honest run locally β record the **video locally**; the Space ships a sample report.
|
| 95 |
+
|
| 96 |
+
## 6. Submission checklist
|
| 97 |
+
- [ ] Gradio app, HF Space under the **org**.
|
| 98 |
+
- [ ] All models <32B; the model <4B; **fine-tuned MiniCPM declared central**.
|
| 99 |
+
- [ ] **Runs fully offline** (no external API β the current OpenBMB call must be replaced) under **llama.cpp** (GGUF).
|
| 100 |
+
- [ ] **Demo video** (local) + **social post** linked in README.
|
| 101 |
+
- [ ] **Public repo** + dense **Codex-attributed commits**.
|
| 102 |
+
- [ ] README states: MiniCPM central, **Modal for fine-tuning**, off-grid, fine-tuned, quantized.
|
| 103 |
+
- [ ] Before/after chart + **traces dataset** + **model card** published.
|
| 104 |
+
- [ ] Space runs without our hardware (sample report bundled).
|
| 105 |
+
|
| 106 |
+
## 7. Prize map (one line each)
|
| 107 |
+
- **OpenAI $10K** β build via Codex (already a contributor); public repo.
|
| 108 |
+
- **OpenBMB $10K** β MiniCPM central (extraction **+ reasoning**); before/after chart.
|
| 109 |
+
- **Modal $20K credits** β LoRA fine-tune (+ eval/data-gen) on Modal.
|
| 110 |
+
- **General pool** β Gradio Space + premium document + video + post.
|
| 111 |
+
- **Badges** β off-grid (local model, no API), fine-tune (LoRA), quantization (GGUF Q4_K_M).
|