| --- |
| license: apache-2.0 |
| base_model: Huangxs/MedPLIB-7b-2e |
| language: |
| - en |
| pipeline_tag: image-segmentation |
| tags: |
| - medical |
| - brain-tumor |
| - mri |
| - segmentation |
| - classification |
| - multimodal |
| - lora |
| - medplib |
| --- |
| |
| # MedPLIB-BRISC |
|
|
| **Joint brain tumor classification and segmentation from MRI with an adapted multimodal large language model.** |
|
|
| This repository holds the weights we trained on top of MedPLIB-7b-2e for the BRISC 2025 brain tumor MRI dataset. Given an MRI slice and a fixed instruction, the model answers with one of four classes and a `<SEG>` token. The four classes are glioma, meningioma, pituitary tumor and non-tumorous. The hidden state of the `<SEG>` token is decoded into the tumor mask, so the class and the mask come from the same response. |
|
|
| [Project page](https://huggingface.co/spaces/Sssunset/MedPLIB-BRISC) · [Code on GitHub](https://github.com/Peilin-FF/Tumor) · [Base model](https://huggingface.co/Huangxs/MedPLIB-7b-2e) |
|
|
|  |
|
|
| ## Files |
|
|
| | File | Content | |
| |---|---| |
| | `trainable.pt` | All trained weights, 286.8M parameters in total: LoRA on the language model, the image projector, the router, the `<SEG>` projector, the SAM-Med2D mask decoder and the SAM-Med2D encoder adapters | |
| | `adapter_meta.json` | LoRA settings and the list of trained tensors | |
| | `assets/` | Figures used on this page | |
|
|
| These weights are not a complete model. They are loaded on top of the original MedPLIB-7b-2e checkpoint, CLIP ViT-L/336 and SAM-Med2D-B with the code from our GitHub repository. |
|
|
| ## How to use |
|
|
| ```bash |
| git clone https://github.com/Peilin-FF/Tumor && cd Tumor |
| scripts/setup_env.sh # conda environment "medplib" |
| scripts/download_models.sh # MedPLIB-7b-2e, CLIP ViT-L/336, SAM-Med2D-B |
| huggingface-cli download Sssunset/MedPLIB-BRISC --local-dir checkpoints/medplib-brisc |
| |
| # interactive demo |
| python -m medplib_bt.demo_app --adapter checkpoints/medplib-brisc --port 7860 |
| |
| # reproduce the test results, which also needs scripts/download_data.sh |
| scripts/run_infer.sh 0,1,2,3 outputs/preds/medplib-brisc --adapter checkpoints/medplib-brisc |
| ``` |
|
|
| ## Training |
|
|
| The model is trained on the 5,000 training slices of the official BRISC 2025 split with 8 NVIDIA A100 GPUs. |
|
|
| | Stage | Epochs | Trained modules | Trainable parameters | Loss | |
| |---|---|---|---|---| |
| | A: Alignment | 1 | image projector, router, `<SEG>` projector, mask decoder | 43.1M, 0.37% | CE + 2 BCE + 0.5 Dice | |
| | B: Adaptation | 12 | Stage A modules, LoRA with rank 16 on attention and both experts in all 32 layers, SAM-Med2D encoder adapters | 286.8M, 2.44% | CE + 2 BCE + 2 Dice + IoU | |
|
|
| Half of the training prompts ask for the class and the mask, a quarter ask for the class only and a quarter ask for the mask only. Non-tumorous slices are trained with an empty mask. At test time, each of the four valid answers is scored by its log-likelihood and the best one is kept. The mask comes from the `<SEG>` state of that answer, and no rule removes it afterwards. |
|
|
|  |
|
|
| ## Results |
|
|
| Results on the 1,000 BRISC test slices, from one training run with seed 42: |
|
|
| | Metric | Value | |
| |---|---| |
| | Accuracy | 0.980 | |
| | Macro-F1 | 0.982 | |
| | Dice on the 860 tumor slices | 0.854 | |
| | Dice on all 1,000 slices | 0.874 | |
| | Masks drawn on the 140 non-tumorous slices | 0 | |
| | Slices where the class and the mask disagree | 0 | |
|
|
| Two separately trained models, EfficientNet-B0 and U-Net, draw a mask on 61.4% of the non-tumorous slices and contradict each other on 9.0% of all slices. Trained on both tasks instead of segmentation alone, the same MedPLIB model stops drawing false masks entirely. The project page has the full comparison and case studies. |
|
|
|  |
|
|
| ## Limitations |
|
|
| - The model is trained and tested on 2D slices from one public dataset. The dataset has no patient identifiers, so slices from one patient may appear in both splits. |
| - A dedicated U-Net still outlines tumors slightly better on average. Gliomas and small lesions are the weakest cases. |
| - Free text answers became very short after adaptation. |
| - This is a course research prototype. It is not a medical device and must not be used for diagnosis or treatment decisions. |
|
|
| ## Team |
|
|
| GP8001 Group 3, Nanyang Technological University: Simon Tong Sing Hee, Zeng Yi, Feng Peilin, Ye Xiaomeng, Lyu Muyang, Timothy Aw Bang Hao. |
|
|
| ## Acknowledgements |
|
|
| This work builds on [MedPLIB](https://arxiv.org/abs/2412.09278), [SAM-Med2D](https://arxiv.org/abs/2308.16184) and the [BRISC 2025 dataset](https://www.nature.com/articles/s41597-026-06753-y). We thank their authors for releasing code, weights and data. |
|
|
