experiments/data_scaling_study/README.md

# Data-Scaling Study — U-Net vs SegFormer-B0

Controlled experiment: **how does training-set size affect the performance of a CNN (U-Net) vs a transformer (SegFormer-B0)** on the solar-panel segmentation task?

| Variable | Values |
|---|---|
| Architecture | U-Net, SegFormer-B0 (`nvidia/mit-b0`) |
| Train data share | 25 %, 50 %, 100 % (nested: 25 ⊂ 50 ⊂ 100) |
| Validation set | full `final_data/val/` (1,331 samples), constant across all 6 runs |
| Image size | 128 × 128 |
| Optimizer | Adam, lr = 1e-4 |
| Scheduler | `ReduceLROnPlateau(mode='max', patience=5, factor=0.5)` on val Dice |
| Loss | `0.5 · BCE + 0.5 · Dice` |
| Augment | HFlip, VFlip, Rot15 |
| Epochs | 50 |
| Batch size | 16 |
| Subset seed | 42 |

The hyperparameters mirror each model's existing trainer in [pv_panel_models/](../../pv_panel_models/) so the only varied factor is **training-data volume**.

## What gets trained vs reused

| Share | U-Net | SegFormer-B0 |
|---|---|---|
| 25 % | trained from scratch | trained from scratch |
| 50 % | trained from scratch | trained from scratch |
| 100 % | **reused** from `pv_panel_models/unet_model/checkpoints/best_model.pth` | **reused** from `pv_panel_models/vit_model/checkpoints/best_model.pth` |

For 100% we copy the existing best checkpoints and parse the existing text logs into JSON. Because the old trainers used per-batch-averaged metrics and never logged mIoU, [bootstrap_100.py](bootstrap_100.py) does a no-grad val pass on each copied checkpoint with the new metrics code so the scaling chart at 100% has a number that's directly comparable to the 25/50% points.

Only **best**-epoch checkpoints are kept (`{model}_{share}_best.pth`). No `_final.pth` is saved.

---

## Layout

```
experiments/data_scaling_study/
├── README.md                 # this file
├── requirements.txt
├── subsets/
│   ├── make_subsets.py       # builds the three subset files (idempotent)
│   ├── subset_25.txt
│   ├── subset_50.txt
│   └── subset_100.txt
├── dataset.py                # SubsetSolarPanelDataset (reads filename list)
├── metrics.py                # confusion-matrix-based mIoU / IoU / Dice / pixel acc
├── models.py                 # re-exports U-Net + SegFormer-B0 builders
├── train.py                  # unified trainer (--model, --share ∈ {25,50})
├── bootstrap_100.py          # copy + parse + val-recompute for 100% point
├── run_all.sh                # runs all 4 trainings + bootstrap
├── checkpoints/              # populated during training / bootstrap
│   ├── unet_25_best.pth        unet_50_best.pth        unet_100_best.pth
│   └── segformer_b0_25_best.pth segformer_b0_50_best.pth segformer_b0_100_best.pth
├── logs/                     # per-run JSON metric history
│   ├── unet_{25,50,100}.json
│   ├── segformer_b0_{25,50,100}.json
│   └── *.stdout.log          # captured stdout from run_all.sh
└── dashboard/
    └── app.py                # Streamlit dashboard
```

---

## How to run

### 0 · Install
```bash
pip install -r requirements.txt
```

### 1 · Build the subsets (once)
```bash
python subsets/make_subsets.py
```
Outputs three text files; asserts `25 ⊂ 50 ⊂ 100`. Re-running is safe and reproducible (seed = 42).

### 2 · Train the four 25/50% runs

One run at a time:
```bash
python train.py --model unet         --share 25
python train.py --model unet         --share 50
python train.py --model segformer_b0 --share 25
python train.py --model segformer_b0 --share 50
```

### 3 · Bootstrap the 100% point
```bash
python bootstrap_100.py            # copy + parse + val-recompute (~30s GPU work)
python bootstrap_100.py --skip-val # if you don't want to run any inference
```

### 2+3 in one go
```bash
./run_all.sh                       # all 4 trainings + bootstrap
./run_all.sh unet                  # only U-Net 25/50% trainings
./run_all.sh segformer_b0          # only SegFormer-B0 25/50% trainings
./run_all.sh bootstrap             # only the bootstrap step
```

Each training run writes:
- `checkpoints/{model}_{share}_best.pth` — highest val Dice across all 50 epochs
- `logs/{model}_{share}.json` — per-epoch loss / dice / iou / miou / pixel_acc

Logs are written incrementally — safe to interrupt and inspect.

The bootstrap step writes:
- `checkpoints/{model}_100_best.pth` — copy of the existing baseline best
- `logs/{model}_100.json` — parsed text-log epochs + `recomputed_val_metrics`

### 4 · Dashboard
```bash
streamlit run dashboard/app.py
```
Three tabs:
1. **Learning curves** — every (model, share) combination, switchable metric, train/val/both. Per-epoch curves at 100% are old-definition (parsed from text log) and mIoU is omitted there.
2. **Data share vs final** — best val mIoU/Dice/IoU/PixelAcc as a function of `% train data`. All three points use the new global metric code (100% via bootstrap).
3. **Inference** — upload an image, see the 2×3 grid of predictions (both models × all shares), with a tweakable threshold.

The dashboard tolerates partial state — you can run it after one training is done and watch the picture fill in.

---

## Metric definitions

We accumulate TP / FP / FN / TN over the full epoch and compute:

| Metric | Formula |
|---|---|
| `iou` (foreground) | TP / (TP + FP + FN) |
| `miou` | mean of foreground-IoU and background-IoU |
| `dice` | 2·TP / (2·TP + FP + FN) |
| `pixel_acc` | (TP + TN) / total |

This differs subtly from the existing baselines in [pv_panel_models/](../../pv_panel_models/), which average per-batch IoU. For a controlled scaling study the global formulation is the standard (and matches PASCAL/Cityscapes-style mIoU reporting).

The 25/50% runs always use the new (global) formulation. The 100% per-epoch curves use the old (per-batch-averaged) formulation parsed from existing text logs — but the **scaling-chart point** at 100% is recomputed under the new formulation by `bootstrap_100.py`'s val pass.

---

## Reproducing the partition

`subsets/make_subsets.py` does:

1. List all `*.jpg` files in `final_data/train/images/` (5,325 files).
2. Sort alphabetically (deterministic order).
3. Shuffle with `random.Random(42)`.
4. Take the first 25 % / 50 % / 100 % → 1,331 / 2,662 / 5,325 files.
5. Assert nesting (25 ⊂ 50 ⊂ 100).

Subsets are stored as plaintext filename lists — both `train.py` and the dashboard read them, so the partition is the single source of truth.