docs/PET_VLM_INNOVATION_ROADMAP.md

# PET-VLM Innovation Roadmap

## 1. What We Should Build

The project should not be framed as ordinary PET classification or segmentation. The current dataset contains paired 3D FDG-PET volumes and regional SUVR tables, but no diagnosis labels, lesion masks, or free-text clinical reports.

The strongest framing is:

> A region-grounded metabolic vision-language framework that converts 3D FDG-PET into structured regional metabolic semantics, and uses this semantic bridge to adapt brain MRI foundation encoders to PET.

Current Stage 1 method name:

> **ReMAP-PET: Region-guided Metabolic Alignment with Partial-tuned PET Encoders**

Short task name:

> PET-to-Regional Metabolic Language Alignment

## 2. Why This Is Different From Existing Brain Foundation Models

Existing brain MRI foundation models mostly learn anatomical or general neuroimaging representations from MRI volumes. NeuroVLM-style work aligns neuroimaging maps with text from publications and activation coordinates. Our data is different:

- The image is FDG-PET, which reflects glucose metabolism rather than anatomy.
- The paired supervision is a 120-region SUVR profile, not a diagnosis label.
- The SUVR table is structured, interpretable, and naturally convertible into language.
- The dataset is small, so full VLM training from scratch is not realistic.

Therefore, the innovation should be a PET-specific semantic adaptation layer over strong pretrained 3D brain encoders.

## 3. Core Model

Recommended architecture:

```text
3D FDG-PET
  -> pretrained 3D brain/medical encoder
  -> PET feature tokens
  -> regional metabolic projector
  -> 120 region tokens
  -> text/SUVR alignment space
  -> retrieval, structured reporting, later VLM/LLM adapter
```

The encoder can be initialized from:

- MedicalNet: stable 3D medical ResNet baseline.
- BrainIAC: general brain MRI foundation model.
- BrainFM: larger brain foundation model, to be integrated after engineering.
- AnatCL: anatomical weak-contrastive MRI model, if weights and code are available.
- Scratch 3D CNN: sanity baseline only.

## 4. Main Innovation: Regional Metabolic Language

Instead of treating the CSV as only regression targets, convert each SUVR table into several aligned representations:

1. Numeric vector:
   - 120-dimensional SUVR values.

2. Region tokens:
   - One token per brain region.
   - Each token contains region identity plus normalized SUVR value.

3. Structured facts:
   - `region = Hippocampus_L`
   - `value = 0.82`
   - `rank = low`
   - `hemisphere = left`
   - `system = medial temporal`

4. Controlled text:
   - "The scan shows relatively low metabolism in bilateral hippocampal and temporal regions."
   - "The highest uptake regions are ..."

This gives us a language-side target without needing real radiology reports.

## 5. Training Objectives

Use multi-task training:

```text
L = L_suvr_regression
  + lambda_1 * L_pet_region_contrastive
  + lambda_2 * L_region_ranking
  + lambda_3 * L_text_alignment
```

### Objective A: PET -> SUVR Regression

Predict the 120-region SUVR vector from the PET image.

Metrics:

- MAE
- RMSE
- Pearson correlation
- Spearman correlation
- per-region error

### Objective B: PET-SUVR Contrastive Alignment

Align each PET embedding with its own SUVR-region embedding.

Metrics:

- PET-to-SUVR Recall@1/5
- SUVR-to-PET Recall@1/5
- MRR
- median rank

### Objective C: Region Ranking

Predict top-k high-metabolism and low-metabolism regions.

Metrics:

- top-k overlap
- high-region F1
- low-region F1
- hemisphere-consistency score

### Objective D: PET-to-Text Alignment

Generate controlled text from the true or predicted SUVR profile, then align PET embeddings with that text.

Metrics:

- PET-to-text retrieval Recall@k
- region mention precision/recall
- factual numeric consistency

## 6. Experimental Matrix

### Backbone Comparison

| Backbone | Frozen Linear | Frozen MLP | Adapter | High-level Partial |
|---|---:|---:|---:|---:|
| Scratch 3D CNN | no | yes | no | yes |
| MedicalNet | yes | yes | yes | yes |
| BrainIAC | yes | yes | yes | yes |
| BrainFM | yes | yes | yes | yes |
| AnatCL | yes | yes | yes | yes |

### Modality Modes

| Mode | Input | Purpose |
|---|---|---|
| A | PET only | Can image encoder recover metabolic structure? |
| B | SUVR/region tokens only | How informative is the structured metabolic profile? |
| C | PET + region/text alignment | Main VLM-style model. |

## 7. Paper-Level Contributions

Contribution 1:

Introduce PET-to-Regional Metabolic Language Alignment, a task that bridges 3D FDG-PET images and interpretable brain-region metabolic semantics.

Contribution 2:

Systematically evaluate whether MRI foundation encoders transfer to FDG-PET metabolic understanding, including frozen probing, adapter tuning, and high-level partial tuning.

Contribution 3:

Propose a region-grounded metabolic tokenizer that converts SUVR tables into numeric, token, fact, and controlled-language supervision.

Contribution 4:

Show that the learned PET encoder supports SUVR prediction, PET-region retrieval, region-level explanation, and future VLM/LLM integration.

## 8. What Not To Claim Yet

Do not claim:

- clinical diagnosis classification, unless diagnosis labels are added;
- segmentation, unless voxel-level masks are added;
- free-form radiology report generation, unless real reports are added;
- full PET foundation model, because 1015 scans are not enough for that claim.

Safer claim:

> PET-specific adaptation of brain foundation encoders through region-grounded metabolic semantic alignment.

## 9. Immediate Next Engineering Steps

1. Add evaluation code for MAE, RMSE, Pearson, Spearman, Recall@k, and top-k region overlap.
2. Add a region-token encoder instead of only using a flat SUVR MLP.
3. Add text template generation from SUVR tables.
4. Add PET-text contrastive training using controlled metabolic summaries.
5. Integrate BrainFM into the same training script.
6. Run final experiments on train/val/test and keep test untouched until final comparison.

## 10. Recommended Title

Primary:

> PET2MetLang: Region-Grounded Metabolic Language Alignment for FDG-PET Brain Representation Learning

Alternative:

> Transferring Brain MRI Foundation Models to FDG-PET via Regional Metabolic Language Alignment