experiments/data_scaling_study/README.md

Data-Scaling Study — U-Net vs SegFormer-B0

Controlled experiment: how does training-set size affect the performance of a CNN (U-Net) vs a transformer (SegFormer-B0) on the solar-panel segmentation task?

Variable	Values
Architecture	U-Net, SegFormer-B0 (nvidia/mit-b0)
Train data share	25 %, 50 %, 100 % (nested: 25 ⊂ 50 ⊂ 100)
Validation set	full final_data/val/ (1,331 samples), constant across all 6 runs
Image size	128 × 128
Optimizer	Adam, lr = 1e-4
Scheduler	ReduceLROnPlateau(mode='max', patience=5, factor=0.5) on val Dice
Loss	0.5 · BCE + 0.5 · Dice
Augment	HFlip, VFlip, Rot15
Epochs	50
Batch size	16
Subset seed	42

The hyperparameters mirror each model's existing trainer in pv_panel_models/ so the only varied factor is training-data volume.

What gets trained vs reused

Share	U-Net	SegFormer-B0
25 %	trained from scratch	trained from scratch
50 %	trained from scratch	trained from scratch
100 %	reused from pv_panel_models/unet_model/checkpoints/best_model.pth	reused from pv_panel_models/vit_model/checkpoints/best_model.pth

For 100% we copy the existing best checkpoints and parse the existing text logs into JSON. Because the old trainers used per-batch-averaged metrics and never logged mIoU, bootstrap_100.py does a no-grad val pass on each copied checkpoint with the new metrics code so the scaling chart at 100% has a number that's directly comparable to the 25/50% points.

Only best-epoch checkpoints are kept ({model}_{share}_best.pth). No _final.pth is saved.

---

Layout

experiments/data_scaling_study/
├── README.md                 # this file
├── requirements.txt
├── subsets/
│   ├── make_subsets.py       # builds the three subset files (idempotent)
│   ├── subset_25.txt
│   ├── subset_50.txt
│   └── subset_100.txt
├── dataset.py                # SubsetSolarPanelDataset (reads filename list)
├── metrics.py                # confusion-matrix-based mIoU / IoU / Dice / pixel acc
├── models.py                 # re-exports U-Net + SegFormer-B0 builders
├── train.py                  # unified trainer (--model, --share ∈ {25,50})
├── bootstrap_100.py          # copy + parse + val-recompute for 100% point
├── run_all.sh                # runs all 4 trainings + bootstrap
├── checkpoints/              # populated during training / bootstrap
│   ├── unet_25_best.pth        unet_50_best.pth        unet_100_best.pth
│   └── segformer_b0_25_best.pth segformer_b0_50_best.pth segformer_b0_100_best.pth
├── logs/                     # per-run JSON metric history
│   ├── unet_{25,50,100}.json
│   ├── segformer_b0_{25,50,100}.json
│   └── *.stdout.log          # captured stdout from run_all.sh
└── dashboard/
    └── app.py                # Streamlit dashboard

---

How to run

0 · Install

pip install -r requirements.txt

1 · Build the subsets (once)

python subsets/make_subsets.py

Outputs three text files; asserts 25 ⊂ 50 ⊂ 100. Re-running is safe and reproducible (seed = 42).

2 · Train the four 25/50% runs

One run at a time:

python train.py --model unet         --share 25
python train.py --model unet         --share 50
python train.py --model segformer_b0 --share 25
python train.py --model segformer_b0 --share 50

3 · Bootstrap the 100% point

python bootstrap_100.py            # copy + parse + val-recompute (~30s GPU work)
python bootstrap_100.py --skip-val # if you don't want to run any inference

2+3 in one go

./run_all.sh                       # all 4 trainings + bootstrap
./run_all.sh unet                  # only U-Net 25/50% trainings
./run_all.sh segformer_b0          # only SegFormer-B0 25/50% trainings
./run_all.sh bootstrap             # only the bootstrap step

Each training run writes:

• checkpoints/{model}_{share}_best.pth — highest val Dice across all 50 epochs
• logs/{model}_{share}.json — per-epoch loss / dice / iou / miou / pixel_acc

Logs are written incrementally — safe to interrupt and inspect.

The bootstrap step writes:

• checkpoints/{model}_100_best.pth — copy of the existing baseline best
• logs/{model}_100.json — parsed text-log epochs + recomputed_val_metrics

4 · Dashboard

streamlit run dashboard/app.py

Three tabs:

1. Learning curves — every (model, share) combination, switchable metric, train/val/both. Per-epoch curves at 100% are old-definition (parsed from text log) and mIoU is omitted there.
2. Data share vs final — best val mIoU/Dice/IoU/PixelAcc as a function of % train data. All three points use the new global metric code (100% via bootstrap).
3. Inference — upload an image, see the 2×3 grid of predictions (both models × all shares), with a tweakable threshold.

The dashboard tolerates partial state — you can run it after one training is done and watch the picture fill in.

---

Metric definitions

We accumulate TP / FP / FN / TN over the full epoch and compute:

Metric	Formula
iou (foreground)	TP / (TP + FP + FN)
miou	mean of foreground-IoU and background-IoU
dice	2·TP / (2·TP + FP + FN)
pixel_acc	(TP + TN) / total

This differs subtly from the existing baselines in pv_panel_models/, which average per-batch IoU. For a controlled scaling study the global formulation is the standard (and matches PASCAL/Cityscapes-style mIoU reporting).

The 25/50% runs always use the new (global) formulation. The 100% per-epoch curves use the old (per-batch-averaged) formulation parsed from existing text logs — but the scaling-chart point at 100% is recomputed under the new formulation by bootstrap_100.py's val pass.

---

Reproducing the partition

subsets/make_subsets.py does:

1. List all *.jpg files in final_data/train/images/ (5,325 files).
2. Sort alphabetically (deterministic order).
3. Shuffle with random.Random(42).
4. Take the first 25 % / 50 % / 100 % → 1,331 / 2,662 / 5,325 files.
5. Assert nesting (25 ⊂ 50 ⊂ 100).

Subsets are stored as plaintext filename lists — both train.py and the dashboard read them, so the partition is the single source of truth.