- <SEG> token. The hidden state at <SEG> is projected and used as the prompt of the SAM-Med2D mask decoder, so the class and the mask are decided together.Overview
-Brain tumor MRI analysis asks two questions: which type of tumor is present, and where it is. These are usually answered by two separate models, a classifier and a segmentation network. Because the two models never share information, their answers can contradict each other. The classifier may report no tumor while the segmentation network still draws one, and this problem stays hidden when each model is only scored on its own task.
-We adapt MedPLIB-7b-2e, a biomedical multimodal large language model, so that a single model answers both questions in one response. On the BRISC 2025 dataset, the adapted model reaches 0.980 accuracy and 0.854 Dice on tumor slices. It never draws a mask on a non-tumorous slice and its class and mask never disagree, without any hand-written rule.
+Brain tumor MRI analysis asks two questions: which type of tumor is present, and where it is. They are usually answered by a classifier and a separate segmentation network that never share information, so their answers can contradict each other. We adapt MedPLIB-7b-2e, a biomedical multimodal large language model, so that one model answers both questions in one response. It writes the tumor type followed by a special <SEG> token, and the hidden state of that token is decoded into the tumor mask. The report and the GitHub README describe the method and the comparison with dedicated models. This page shows what they leave out.
Method
-A unified model
-A CLIP vision encoder and an image projector turn the MRI slice \(x\) into tokens for a Llama language model with two expert branches and a router. Given a fixed instruction \(q\), the model writes an answer of the form “\(c\) <SEG>”, where \(c\) names one of four classes: glioma, meningioma, pituitary tumor or non-tumorous. The hidden state at the <SEG> position is projected to a prompt vector \(e\) and combined with the SAM-Med2D image features \(F\) to give the mask:
-Inference by answer scoring
-Only four answers are valid, so we do not parse free text. Each candidate answer is scored by its log-likelihood and the best one is kept. The mask is decoded from the <SEG> state of the winning answer, and no rule removes it afterwards. An empty mask on a non-tumorous slice therefore has to come from the model itself.
-Two-stage adaptation
-Stage A trains only the small modules that connect the vision encoders, the language model and the mask decoder. Stage B adds LoRA with rank 16 to the attention and both experts in all 32 language model layers, and switches on the adapter layers of the SAM-Med2D encoder. Half of the training prompts ask for the class and the mask, a quarter ask for the class only and a quarter ask for the mask only. Non-tumorous slices are trained with an empty mask.
+Ask the model
+Pick a test slice. The model scores each of the four valid answers, keeps the most likely one, and draws the mask from that answer's <SEG> token.
+
- Results
-All models are trained on the official BRISC 2025 split, 5,000 training and 1,000 test slices, and scored by the same evaluation script. Of the test slices, 860 contain a tumor and 140 do not. The unified model results come from one training run with seed 42.
+Training is short and stable
+Both stages together take about two hours on 8 NVIDIA A100 GPUs.
+
+ | Model | Accuracy | Macro-F1 | Tumor Dice | Dice, all slices | False mask rate | Contradictions | Both answers correct |
|---|---|---|---|---|---|---|---|
| MedPLIB, no adaptation | 0.294 | 0.123 | 0.040 | 0.085 | 63.6% | 7.0% | 1.2% |
| MedPLIB, unified, ours | 0.980 | 0.982 | 0.854 | 0.874 | 0.0% | 0.0% | 93.9% |
| EfficientNet-B0 | 0.993 | 0.994 | – | – | – | – | – |
| U-Net, ResNet-34 | – | – | 0.881 | 0.812 | 61.4% | – | – |
| nnU-Net 2D | – | – | 0.877 | 0.807 | 62.1% | – | – |
| EfficientNet-B0 + U-Net | 0.993 | 0.994 | 0.881 | 0.812 | 61.4% | 9.0% | 87.0% |
| EfficientNet-B0 + U-Net, checked | 0.993 | 0.994 | 0.881 | 0.898 | 0.0% | 0.4% | 95.6% |
| EfficientNet-B0 + nnU-Net, checked | 0.993 | 0.994 | 0.877 | 0.894 | 0.0% | 0.5% | 95.3% |
Tumor Dice is computed on the 860 tumor slices. Dice over all slices also counts the 140 non-tumorous slices, where an empty mask scores 1 and any mask scores 0. The false mask rate is the share of non-tumorous slices that receive a mask. Contradictions count slices where the class and the mask disagree about whether a tumor is present. “Checked” means that a hand-written rule keeps the segmentation mask only if EfficientNet-B0 predicts a tumor.
- -
- Joint training teaches the mask when to stay empty
-To isolate the effect of training both tasks in one model, we trained the same model with the same shorter setting on one task at a time. Trained for segmentation alone, MedPLIB draws a mask on 27.9% of non-tumorous slices. Trained on both tasks, it draws none, and its Dice rises while its accuracy stays almost the same.
+What adaptation changes
+The same model on the same 1,000 test slices, before and after our two training stages.
+
+ Real answers before and after adaptation
+The first two cases show the gain from adaptation. The third shows its cost: training on one narrow task made the model lose some general skills, and its free descriptions shrank to a single word. Mixing general question and answer data into training is one way to keep these skills.
| Training task | Accuracy | Tumor Dice | Dice, all slices | False mask rate |
|---|---|---|---|---|
| Classification only | 0.968 | – | – | – |
| Segmentation only | – | 0.737 | 0.735 | 27.9% |
| Both tasks, unified | 0.964 | 0.751 | 0.786 | 0.0% |
All three runs use 6 epochs of Stage B with the Stage A loss and seed 42, so they differ only in the training task.
Where the model is still weakest
-Tumor Dice of the unified model is 0.763 on gliomas, 0.919 on meningiomas and 0.865 on pituitary tumors. It is 0.728 on small lesions and 0.892 on large ones. All 20 classification errors are confusions between tumor types, never between tumor and no tumor. With the hand-written rule, the EfficientNet-B0 and U-Net combination still reaches a slightly higher Dice over all slices, mainly because U-Net outlines tumors more precisely. Closing this gap is our next step.
+The model knows when it is unsure
+The four answer scores give a confidence for every prediction, at no extra cost.
+
+ Case comparisons
-In every panel, the green outline is the expert annotation and the red area is the predicted mask. A good result is a red area that fills the green outline. Red text marks a wrong answer. We selected these cases to show where the unified model helps.
+Results at a glance
+| Model | Accuracy | Tumor Dice | Dice, all slices | False masks | Contradictions |
|---|---|---|---|---|---|
| MedPLIB, no adaptation | 0.294 | 0.040 | 0.085 | 63.6% | 7.0% |
| MedPLIB, unified, ours | 0.980 | 0.854 | 0.874 | 0.0% | 0.0% |
| EfficientNet-B0 + U-Net | 0.993 | 0.881 | 0.812 | 61.4% | 9.0% |
| EfficientNet-B0 + U-Net, with a hand-written rule | 0.993 | 0.881 | 0.898 | 0.0% | 0.4% |
Dice over all slices also scores the 140 non-tumorous slices, where an empty mask counts as correct. The hand-written rule removes the U-Net mask whenever EfficientNet-B0 predicts no tumor. Accuracy of the unified model is about the same on all three imaging planes: 0.980 on axial, 0.984 on coronal and 0.976 on sagittal slices. The full comparison is in the README.
- 
- Case gallery
+For each tumor type, the test slices where the unified model scored its best, median and worst Dice, and the two non-tumorous slices with the largest false masks before adaptation. Nothing is hand picked.
+ +Limitations
- The model is trained and tested on 2D slices from one public dataset. The dataset has no patient identifiers, so slices from one patient may appear in both splits.
- Results come from a single training run. We plan to repeat training with more random seeds. -
- Free text answers became very short after adaptation. +
- A dedicated U-Net still outlines tumors slightly better on average, and gliomas and small lesions are the hardest cases.
- This is a course research prototype and must not be used for clinical decisions.
This work builds on MedPLIB, SAM-Med2D and the BRISC 2025 dataset. We thank their authors for releasing code, weights and data.