| --- |
| license: cc-by-nc-4.0 |
| language: |
| - en |
| - tr |
| pipeline_tag: image-text-to-text |
| library_name: peft |
| tags: |
| - early_diagnosis |
| - reasoning |
| - diagnosis |
| - health |
| - healthcare |
| - alzheimer |
| - athropy |
| - dementia |
| - biomarkers |
| - biology |
| - academic |
| - peft |
| - lora |
| - mri |
| --- |
| |
| # VLbai-2.6AD |
|
|
| A clinical reasoning assistant for early-stage Alzheimer's assessment. It joins a |
| 3D MRI + biomarker classifier (**Vbai-2.6AD**) to a reasoning LLM (**Gemma 4 12B**) |
| inside a single forward pass — the diagnosis is passed as **vectors, not text**. |
|
|
| Answers are in Turkish. |
|
|
| > **Research use only.** Not a medical device. Not validated for clinical |
| > decision-making. See [Limitations](#limitations). |
|
|
| --- |
|
|
| ## What This Is (and is not) |
|
|
| Most "LLM + medical model" systems are pipelines: the classifier prints a label, |
| the label is pasted into a prompt, the LLM writes prose around it. The LLM never |
| sees the evidence — only a word. |
|
|
| Here, Vbai-2.6AD's fused representation (1280-d) is projected into 4 soft tokens that |
| live in Gemma's embedding space and are marked as non-text via |
| `mm_token_type_ids=1`. Gemma attends to the *representation*, not to a string. |
|
|
| ``` |
| 3D MRI (T1 96³) 13 Biomark + Masks |
| │ │ |
| ResNet3D + CBAM/SE/ASPP TabularEncoder |
| │ │ |
| └─────── CrossModalFusion ───────┐ |
| │ │ |
| Combined Vector (1280-d) Classification Header |
| │ (CN / MCI / AD & 5-Year Risk) |
| Projector (LN→Dense...) │ |
| │ │ |
| 4 Soft Token (3840-d) │ |
| │ │ |
| [Text Embedding] ++ [SOFT TOKENS] ++ [Text Embedding] ──► Gemma 12B + LoRA |
| │ |
| Turkish Clinical Description |
| ``` |
|
|
| The classifier's verdict is an **anchor**: the LLM explains and defends it, but |
| cannot overturn it. Numbers come from code, prose comes from the LLM. |
|
|
| ### Evidence That the Vector Channel is Real |
|
|
| Zeroing the soft tokens while keeping the text prompt **identical** changes the |
| stated class in **47.5%** of cases (19/40). If this were a pipeline dressed up as |
| fusion, the text alone would carry the answer and ablation would change nothing. |
|
|
| --- |
|
|
| ## Results |
|
|
| Evaluated on a held-out test split (40 unique patients, `eval_faithfulness.py`). |
|
|
| | Metric | Value | What it means | |
| |---|---|---| |
| | Faithfulness to head verdict | **97.5%** (39/40) | LLM does not invent its own diagnosis | |
| | Agreement with ground truth | **92.5%** | Matches Vbai-2.6AD's own accuracy (0.919) — the LLM layer neither helps nor harms | |
| | Soft-token ablation | **47.5%** | The image/fusion channel is genuinely read | |
| | Ambiguous outputs | 0/40 | Always produced a parseable verdict | |
|
|
| ## Classification and Early Risk Model General Tests (Only Vbai-2.6AD) |
|
|
| | Input/Patch Size | Params | Accuracy | ROC-AUC | F1 Score | F1 Score (Median) | Recall | Precision | F2 Score | MCC | Specificity | FPR | FNR | |
| |:-------:|:-------:|:---------:|:---------:|:---------:|:--------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:| |
| | _96³_ + T1 + 13 Biomarkers (Optional) | 16.85M | %86.67 | %96.48 | %86.30 | %85.71 | %86.67 | %88.39 | %86.24 | %81.20 | %93.33 | %6.66 | %13.33 | |
|
|
| | Class | Preicision | Recall | F1 Score | Support | |
| |:-:|:-:|:-:|:-:|:-:| |
| | CN | %97.94 | %95.00 | %96.45 | 100 | |
| | MCI | %91.67 | %66.00 | %76.74 | 100 | |
| | AD | %75.57 | %99.00 | %85.71 | 100 | |
|
|
| Ablation of the classifier itself, before any LLM involvement: |
|
|
| | Input | Accuracy | |
| |---|---| |
| | Fusion (MRI + biomarkers) | 0.895 | |
| | Biomarkers only | 0.871 | |
| | MRI only | 0.448 | |
|
|
| MRI changes 20/123 predictions relative to biomarkers alone (14 better, 6 worse). |
|
|
| ***Tested with ADNI T1 and ADNIMERGE values datasets. But training is excluding ADNI T1 and ADNIMERGE values datasets.** |
| |
| ***It was trained in just 14 epochs.** |
|
|
| ***No transfer learning or pre-trained weights were used.** |
| |
| ### Catastrophic Forgetting |
| |
| LoRA adapts the model to a narrow task. These benchmarks measure how much general |
| biomedical reasoning was lost — they are **not** a measure of this model's |
| clinical ability. |
|
|
| | Benchmark | n | Base | + LoRA | Δ | |
| |---|---|---|---|---| |
| | MedMCQA (validation, single-choice) | 300 | 0.557 | 0.570 | +0.013 | |
| | PubMedQA (pqa_labeled) | 300 | 0.710 | 0.697 | −0.013 | |
| |
| Equal magnitude in opposite directions, both inside the ±0.029 standard error at |
| n=300. **No measurable forgetting.** We attribute this to the multitask training |
| data (8 task types) rather than to a single templated objective. |
| |
| ### Run-to-Run Variance |
| |
| LoRA training here is **not** stable across seeds. On a later data revision we |
| trained three seeds and observed: |
| |
| | Seed | val loss | Faithfulness | Ground truth | Ablation | |
| |---|---|---|---|---| |
| | unseeded | 0.0278 | 80.0% | 75.0% | 32.5% | |
| | 1 | 0.0246 | 97.5% | 92.5% | 75.0% | |
| | 2 | 0.0226 | 100% | 95.0% | 55.0% | |
| |
| Validation loss barely moved while behaviour swung by 20 points. **The released |
| checkpoint is a single run**; treat its numbers as one draw from this |
| distribution, not as a stable expectation. `--seed` is exposed in |
| `train_projector.py` for reproduction. |
|
|
| --- |
|
|
| ## Usage |
|
|
| Load Gemma yourself, then apply the adapter and projector. Merged weights are |
| deliberately **not** distributed (see [License](#license)). |
|
|
| ```bash |
| python chat.py \ |
| --features vlbai-2.6ad_features_tbm.pt \ |
| --text vlbai-2.6ad_text.json \ |
| --projector projector.pt |
| ``` |
|
|
| Two modes: |
|
|
| - **`report`** — thinking off, short answers, questions chosen from a menu of 8 |
| canonical questions (`/questions`). Free-typed questions are allowed but warned |
| about. |
| - **`discuss`** — thinking on, long-form reasoning, free input. |
|
|
| ### Why the menu exists |
|
|
| LoRA learns *phrase patterns*, not task boundaries. A free-typed question that |
| falls between trained patterns gets answered with a blend of them, which |
| occasionally produced self-contradictory text ("atrophy is marked" and "all |
| values normal" in one answer). Restricting `rapor` mode to in-distribution |
| questions removes the failure mechanism rather than patching its symptoms. |
|
|
| --- |
|
|
| ## Training data |
|
|
| 817 ADNI patients → 7353 supervised examples across 8 task types: |
|
|
| | Task | Purpose | |
| |---|---| |
| | `cls` | Classification + progression risk | |
| | `region` | Which regions show atrophy (18 FreeSurfer ROIs, ICV-normalized) | |
| | `amyloid` | Amyloid status via ATN framework (NIA-AA 2018) | |
| | `missing` | Which biomarkers were not measured | |
| | `hold` / `hold2` | Hold position under clinician pushback | |
| | `absent` | Refuse to comment on data not in the panel | |
| | `probs` | Report exact class probabilities and risk | |
|
|
| **Every target is rule-derived from the data — none are LLM-synthesized.** This |
| is deliberate: a synthesized target teaches the model to sound confident about |
| numbers it did not compute. Digits appear in targets only where the code owns |
| them (`probs`, ROI z-scores). |
|
|
| The `absent` task exists because of a real failure: when a clinician said "there |
| is a family history of early-onset Alzheimer's", an earlier model replied that |
| "the model assessment includes these findings". It did not. Accepting a false |
| premise and reasoning on top of it is the most dangerous error class in a |
| clinical tool. |
|
|
| Region measurements come from ADNI's UCSFFSX7 FreeSurfer table, ICV-normalized, |
| region codes resolved by name from `DATADIC.rda`. 811/817 patients matched with a |
| median scan-to-table interval of 0 days. |
|
|
| **Generated datasets are not redistributed** — they are ADNI-derived. The |
| build scripts are included so holders of ADNI access can regenerate them. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **Single cohort.** 817 ADNI patients. No external validation. Performance on |
| other scanners, populations, or preprocessing pipelines is unknown. |
| - **Modality is not interchangeable.** This checkpoint was trained on TBM |
| Jacobian volumes. Feeding raw T1 produces confident nonsense. The extraction |
| scripts require an explicit `--tbm` / `--t1` flag for this reason. |
| - **The `absent` task can undermine its own anchor.** In this checkpoint, the |
| closing sentence of `absent` targets ends on the word "assessment", and the |
| model has been observed to blend it with the `hold` task into |
| *"the Vbai-2.6AD assessment can be changed if clinically meaningful"* — which |
| contradicts the anchor rule. `rapor` mode's menu prevents this in practice; |
| a caller bypassing the menu is not protected. Fixed in the data generator |
| after this checkpoint was trained. |
| - **One patient reproducibly breaks.** `005_S_0324` produced corrupted output |
| (stray non-Turkish tokens, leaked control text) in 4/4 training runs. Suspected |
| outlier feature vector. Not yet diagnosed. |
| - **Single training run.** See the variance table above. |
| - **Turkish only.** Prompts and targets are Turkish; other languages are |
| untested. |
| - **Not a medical device.** No regulatory clearance. Research use only. |
|
|
| --- |
|
|
| ## Evaluation scripts |
|
|
| There is no accepted benchmark for "MRI-grounded Alzheimer's assistant". These |
| scripts fill that gap and are included so results can be reproduced or disputed: |
|
|
| | Script | Measures | |
| |---|---| |
| | `eval_faithfulness.py` | Anchor fidelity + soft-token ablation | |
| | `eval_interaction.py` | Grounding, missing data, position holding | |
| | `eval_thinking.py` | Open-ended reasoning, thinking on vs off | |
| | `benchmark_medical.py` | MedMCQA / PubMedQA, base vs LoRA | |
| | `probe_features.py` | Is the information linearly decodable | |
| | `mri_contribution.py` | MRI's contribution to the decision | |
|
|
| --- |
|
|
| ## License |
|
|
| Two layers, both apply: |
|
|
| 1. **Projector, training code, evaluation scripts, data generators** — original |
| work, released under **CC-BY-NC-4.0**. |
| 2. **LoRA adapter** — a derivative of Gemma 4. Google's **Gemma Terms of Use** |
| govern it and are not overridden by the license above. You must comply with |
| both. |
|
|
| Base model weights are not redistributed. Download Gemma from Google under its |
| own terms. |
|
|
| **ADNI.** These weights encode information derived from ADNI data. Use is subject |
| to the ADNI Data Use Agreement. Redistribution of ADNI-derived datasets is not |
| permitted here; generated training files are excluded from this repository for |
| that reason. |
|
|
| ## Acknowledgements |
|
|
| Data collection and sharing for this project was funded by the Alzheimer's |
| Disease Neuroimaging Initiative (ADNI). ADNI investigators contributed to the |
| design and implementation of ADNI and provided data but did not participate in |
| the analysis or writing of this work. |
|
|
| ## Requirements |
|
|
| - Python ≥ 3.9 |
| - PyTorch ≥ 2.0 |
| - CUDA-capable GPU, ≥ 16 GB VRAM recommended (Tested with at least an NVIDIA L4 GPU with 24 GB of VRAM) (Trained with NVIDIA A100 with of 40 GB of VRAM) |
| - See `requirements.txt` for full dependency list |
|
|
| ### Support |
|
|
| - **Website**: [Neurazum](https://neurazum.com) |
| - **Email**: [contact@neurazum.com](mailto:contact@neurazum.com) |
|
|
| --- |
|
|
