docs/PET_VLM_INNOVATION_ROADMAP.md

PET-VLM Innovation Roadmap

1. What We Should Build

The project should not be framed as ordinary PET classification or segmentation. The current dataset contains paired 3D FDG-PET volumes and regional SUVR tables, but no diagnosis labels, lesion masks, or free-text clinical reports.

The strongest framing is:

A region-grounded metabolic vision-language framework that converts 3D FDG-PET into structured regional metabolic semantics, and uses this semantic bridge to adapt brain MRI foundation encoders to PET.

Current Stage 1 method name:

ReMAP-PET: Region-guided Metabolic Alignment with Partial-tuned PET Encoders

Short task name:

PET-to-Regional Metabolic Language Alignment

2. Why This Is Different From Existing Brain Foundation Models

Existing brain MRI foundation models mostly learn anatomical or general neuroimaging representations from MRI volumes. NeuroVLM-style work aligns neuroimaging maps with text from publications and activation coordinates. Our data is different:

• The image is FDG-PET, which reflects glucose metabolism rather than anatomy.
• The paired supervision is a 120-region SUVR profile, not a diagnosis label.
• The SUVR table is structured, interpretable, and naturally convertible into language.
• The dataset is small, so full VLM training from scratch is not realistic.

Therefore, the innovation should be a PET-specific semantic adaptation layer over strong pretrained 3D brain encoders.

3. Core Model

Recommended architecture:

3D FDG-PET
  -> pretrained 3D brain/medical encoder
  -> PET feature tokens
  -> regional metabolic projector
  -> 120 region tokens
  -> text/SUVR alignment space
  -> retrieval, structured reporting, later VLM/LLM adapter

The encoder can be initialized from:

• MedicalNet: stable 3D medical ResNet baseline.
• BrainIAC: general brain MRI foundation model.
• BrainFM: larger brain foundation model, to be integrated after engineering.
• AnatCL: anatomical weak-contrastive MRI model, if weights and code are available.
• Scratch 3D CNN: sanity baseline only.

4. Main Innovation: Regional Metabolic Language

Instead of treating the CSV as only regression targets, convert each SUVR table into several aligned representations:

1. Numeric vector:

   • 120-dimensional SUVR values.

2. Region tokens:

   • One token per brain region.
   • Each token contains region identity plus normalized SUVR value.

3. Structured facts:

   • region = Hippocampus_L
   • value = 0.82
   • rank = low
   • hemisphere = left
   • system = medial temporal

4. Controlled text:

   • "The scan shows relatively low metabolism in bilateral hippocampal and temporal regions."
   • "The highest uptake regions are ..."

This gives us a language-side target without needing real radiology reports.

5. Training Objectives

Use multi-task training:

L = L_suvr_regression
  + lambda_1 * L_pet_region_contrastive
  + lambda_2 * L_region_ranking
  + lambda_3 * L_text_alignment

Objective A: PET -> SUVR Regression

Predict the 120-region SUVR vector from the PET image.

Metrics:

• MAE
• RMSE
• Pearson correlation
• Spearman correlation
• per-region error

Objective B: PET-SUVR Contrastive Alignment

Align each PET embedding with its own SUVR-region embedding.

Metrics:

• PET-to-SUVR Recall@1/5
• SUVR-to-PET Recall@1/5
• MRR
• median rank

Objective C: Region Ranking

Predict top-k high-metabolism and low-metabolism regions.

Metrics:

• top-k overlap
• high-region F1
• low-region F1
• hemisphere-consistency score

Objective D: PET-to-Text Alignment

Generate controlled text from the true or predicted SUVR profile, then align PET embeddings with that text.

Metrics:

• PET-to-text retrieval Recall@k
• region mention precision/recall
• factual numeric consistency

6. Experimental Matrix

Backbone Comparison

Backbone	Frozen Linear	Frozen MLP	Adapter	High-level Partial
Scratch 3D CNN	no	yes	no	yes
MedicalNet	yes	yes	yes	yes
BrainIAC	yes	yes	yes	yes
BrainFM	yes	yes	yes	yes
AnatCL	yes	yes	yes	yes

Modality Modes

Mode	Input	Purpose
A	PET only	Can image encoder recover metabolic structure?
B	SUVR/region tokens only	How informative is the structured metabolic profile?
C	PET + region/text alignment	Main VLM-style model.

7. Paper-Level Contributions

Contribution 1:

Introduce PET-to-Regional Metabolic Language Alignment, a task that bridges 3D FDG-PET images and interpretable brain-region metabolic semantics.

Contribution 2:

Systematically evaluate whether MRI foundation encoders transfer to FDG-PET metabolic understanding, including frozen probing, adapter tuning, and high-level partial tuning.

Contribution 3:

Propose a region-grounded metabolic tokenizer that converts SUVR tables into numeric, token, fact, and controlled-language supervision.

Contribution 4:

Show that the learned PET encoder supports SUVR prediction, PET-region retrieval, region-level explanation, and future VLM/LLM integration.

8. What Not To Claim Yet

Do not claim:

• clinical diagnosis classification, unless diagnosis labels are added;
• segmentation, unless voxel-level masks are added;
• free-form radiology report generation, unless real reports are added;
• full PET foundation model, because 1015 scans are not enough for that claim.

Safer claim:

PET-specific adaptation of brain foundation encoders through region-grounded metabolic semantic alignment.

9. Immediate Next Engineering Steps

1. Add evaluation code for MAE, RMSE, Pearson, Spearman, Recall@k, and top-k region overlap.
2. Add a region-token encoder instead of only using a flat SUVR MLP.
3. Add text template generation from SUVR tables.
4. Add PET-text contrastive training using controlled metabolic summaries.
5. Integrate BrainFM into the same training script.
6. Run final experiments on train/val/test and keep test untouched until final comparison.

10. Recommended Title

Primary:

PET2MetLang: Region-Grounded Metabolic Language Alignment for FDG-PET Brain Representation Learning

Alternative:

Transferring Brain MRI Foundation Models to FDG-PET via Regional Metabolic Language Alignment