README.md

---
license: apache-2.0
base_model: Huangxs/MedPLIB-7b-2e
language:
- en
pipeline_tag: image-segmentation
tags:
- medical
- brain-tumor
- mri
- segmentation
- classification
- multimodal
- lora
- medplib
---

# MedPLIB-BRISC

**Joint brain tumor classification and segmentation from MRI with an adapted multimodal large language model.**

This repository holds the weights we trained on top of MedPLIB-7b-2e for the BRISC 2025 brain tumor MRI dataset. Given an MRI slice and a fixed instruction, the model answers with one of four classes and a `<SEG>` token. The four classes are glioma, meningioma, pituitary tumor and non-tumorous. The hidden state of the `<SEG>` token is decoded into the tumor mask, so the class and the mask come from the same response.

[Project page](https://huggingface.co/spaces/Sssunset/MedPLIB-BRISC) · [Code on GitHub](https://github.com/Peilin-FF/Tumor) · [Base model](https://huggingface.co/Huangxs/MedPLIB-7b-2e)

![Model overview](assets/fig_architecture.png)

## Files

| File | Content |
|---|---|
| `trainable.pt` | All trained weights, 286.8M parameters in total: LoRA on the language model, the image projector, the router, the `<SEG>` projector, the SAM-Med2D mask decoder and the SAM-Med2D encoder adapters |
| `adapter_meta.json` | LoRA settings and the list of trained tensors |
| `assets/` | Figures used on this page |

These weights are not a complete model. They are loaded on top of the original MedPLIB-7b-2e checkpoint, CLIP ViT-L/336 and SAM-Med2D-B with the code from our GitHub repository.

## How to use

```bash
git clone https://github.com/Peilin-FF/Tumor && cd Tumor
scripts/setup_env.sh                      # conda environment "medplib"
scripts/download_models.sh                # MedPLIB-7b-2e, CLIP ViT-L/336, SAM-Med2D-B
huggingface-cli download Sssunset/MedPLIB-BRISC --local-dir checkpoints/medplib-brisc

# interactive demo
python -m medplib_bt.demo_app --adapter checkpoints/medplib-brisc --port 7860

# reproduce the test results, which also needs scripts/download_data.sh
scripts/run_infer.sh 0,1,2,3 outputs/preds/medplib-brisc --adapter checkpoints/medplib-brisc
```

## Training

The model is trained on the 5,000 training slices of the official BRISC 2025 split with 8 NVIDIA A100 GPUs.

| Stage | Epochs | Trained modules | Trainable parameters | Loss |
|---|---|---|---|---|
| A: Alignment | 1 | image projector, router, `<SEG>` projector, mask decoder | 43.1M, 0.37% | CE + 2 BCE + 0.5 Dice |
| B: Adaptation | 12 | Stage A modules, LoRA with rank 16 on attention and both experts in all 32 layers, SAM-Med2D encoder adapters | 286.8M, 2.44% | CE + 2 BCE + 2 Dice + IoU |

Half of the training prompts ask for the class and the mask, a quarter ask for the class only and a quarter ask for the mask only. Non-tumorous slices are trained with an empty mask. At test time, each of the four valid answers is scored by its log-likelihood and the best one is kept. The mask comes from the `<SEG>` state of that answer, and no rule removes it afterwards.

![Two training stages](assets/fig_stages.png)

## Results

Results on the 1,000 BRISC test slices, from one training run with seed 42:

| Metric | Value |
|---|---|
| Accuracy | 0.980 |
| Macro-F1 | 0.982 |
| Dice on the 860 tumor slices | 0.854 |
| Dice on all 1,000 slices | 0.874 |
| Masks drawn on the 140 non-tumorous slices | 0 |
| Slices where the class and the mask disagree | 0 |

Two separately trained models, EfficientNet-B0 and U-Net, draw a mask on 61.4% of the non-tumorous slices and contradict each other on 9.0% of all slices. Trained on both tasks instead of segmentation alone, the same MedPLIB model stops drawing false masks entirely. The project page has the full comparison and case studies.

![Results](assets/fig_results.png)

## Limitations

- The model is trained and tested on 2D slices from one public dataset. The dataset has no patient identifiers, so slices from one patient may appear in both splits.
- A dedicated U-Net still outlines tumors slightly better on average. Gliomas and small lesions are the weakest cases.
- Free text answers became very short after adaptation.
- This is a course research prototype. It is not a medical device and must not be used for diagnosis or treatment decisions.

## Team

GP8001 Group 3, Nanyang Technological University: Simon Tong Sing Hee, Zeng Yi, Feng Peilin, Ye Xiaomeng, Lyu Muyang, Timothy Aw Bang Hao.

## Acknowledgements

This work builds on [MedPLIB](https://arxiv.org/abs/2412.09278), [SAM-Med2D](https://arxiv.org/abs/2308.16184) and the [BRISC 2025 dataset](https://www.nature.com/articles/s41597-026-06753-y). We thank their authors for releasing code, weights and data.