Tao-AI-Informatics/SAM3-LoRA-Breast-Lesion

SAM3 LoRA Breast Lesion

Portable SAM3 semantic-text segmentation bundle fine-tuned on the BUSCoT trainval split. The training prompt was breast lesion.

Important Architecture Note

Inference requires two checkpoints:

1. model/sam3_base.pt: original SAM3 base checkpoint used to construct SAM3.
2. model/best_model.pt: fine-tuned LoRA/native-head checkpoint.

best_model.pt is not a self-contained model. The bundled wrapper first builds the original SAM3 architecture from sam3_base.pt, injects LoRA modules, and then loads best_model.pt.

The bundle also includes:

• runtime/sam3_repo/sam3/: required SAM3 Python source.
• scripts/sam3_decoder_experiment_lib.py: custom LoRA/native-decoder model.
• scripts/sam3_buscot_runner.py: single-image prediction wrapper.

No paths from the original training server are required for inference.

Model Configuration

Item	Value
Prompt type	Semantic text
Training prompt	breast lesion
Input size	512
LoRA rank	8
LoRA alpha	16
Default threshold	0.5
Validation-selected threshold	0.6

BUSCoT test performance at threshold 0.5:

Mean Dice	Mean IoU	Mean HD95	Mean ASSD
0.8782	0.8130	27.35	10.57

Package Layout

model/
  sam3_base.pt                 original SAM3 base checkpoint
  best_model.pt                fine-tuned checkpoint
  config.json
  thresholds.json
runtime/sam3_repo/
  sam3/                        SAM3 source and tokenizer asset
scripts/
  infer_single_image.py        portable single-image inference
  sam3_buscot_runner.py
  sam3_decoder_experiment_lib.py
  evaluate_sam3_test_text_prompt_segmentation.py
  evaluate_sam3_lora_multiprompt_sensitivity.py
  analyze_sam3_prompt_robust_vs_nonrobust.py
  train_sam3_decoder_segmentation.py
  run_single_prompt_eval.sh
  run_5prompt_qc.sh
metrics/
examples/
requirements.txt

Environment

Python 3.10 is recommended. Install dependencies into an environment that has a CUDA-compatible PyTorch build:

cd "/path/to/SAM3 LoRA Breast Lesion"
python -m pip install -r requirements.txt

If a compatible SAM3 environment already exists, activate it and skip the installation command.

Single-Image Inference

Run from any directory:

CUDA_VISIBLE_DEVICES=0 python \
  "/path/to/SAM3 LoRA Breast Lesion/scripts/infer_single_image.py" \
  --image /path/to/breast_ultrasound.png \
  --output /path/to/predicted_mask.png \
  --prompt "breast lesion"

This inference path does not use a GT mask or bbox.

To use an external SAM3 base checkpoint instead of the bundled copy:

SAM3_CHECKPOINT=/path/to/sam3.pt CUDA_VISIBLE_DEVICES=0 python \
  "/path/to/SAM3 LoRA Breast Lesion/scripts/infer_single_image.py" \
  --image /path/to/image.png \
  --output /path/to/mask.png

Batch Evaluation With GT

The input CSV must contain at least:

sample_id,original_image_path,gt_mask_path,gt_mask_bbox,has_original_image,has_gt_mask

Run:

cd "/path/to/SAM3 LoRA Breast Lesion"
PYTHON=/path/to/python bash scripts/run_single_prompt_eval.sh \
  /path/to/index.csv /path/to/output 0

Arguments are INDEX_CSV, OUTPUT_DIR, and GPU.

Five-Prompt QC

cd "/path/to/SAM3 LoRA Breast Lesion"
PYTHON=/path/to/python bash scripts/run_5prompt_qc.sh \
  /path/to/index.csv /path/to/output 0

The prompts are:

breast lesion
breast tumor
breast mass
breast nodule
ultrasound breast lesion

The QC pipeline calculates pairwise predicted-mask Dice and uses minimum pairwise Dice as a deployable consistency signal. GT masks are needed only for offline accuracy evaluation.

Optional Retraining

Retraining requires a compatible dataset CSV with train, val, and test rows. It is not needed for inference:

cd "/path/to/SAM3 LoRA Breast Lesion"
PYTHON=/path/to/python BATCH_SIZE=8 bash scripts/train_sam3_buscot_lora.sh \
  /path/to/dataset.csv /path/to/new_output 0

Troubleshooting

Verify all bundled files:

cd "/path/to/SAM3 LoRA Breast Lesion"
sha256sum -c SHA256SUMS

Verify imports without loading the checkpoints:

cd /tmp
PYTHONPATH="/path/to/SAM3 LoRA Breast Lesion/scripts:/path/to/SAM3 LoRA Breast Lesion/runtime/sam3_repo" \
python -c "from sam3_buscot_runner import SAM3BuscotPredictor; print('imports OK')"

Common issues:

• No module named sam3_decoder_experiment_lib: bundle is incomplete or an old copy is being used.
• No module named sam3: runtime/sam3_repo/sam3/ is missing.
• Missing sam3_base.pt: the original base checkpoint was not transferred.
• CUDA/PyTorch errors: install a PyTorch build compatible with the target server's CUDA driver.
• Package version errors: use the provided requirements.txt and Python 3.10.

License

The bundled SAM3 source and base checkpoint remain subject to the SAM3 license included at runtime/sam3_repo/LICENSE.