isaaccorley/ftw-prue-ccby

Runs using the PRUE EfficientNet configs but with training w/o only on CC-BY countries.

Metrics were computed on the FTW Full dataset using ftw model test -p3 -t3 --temporal_options stacked --model path/to/model.ckpt --countries full_data

Model	Pixel Level IoU	Pixel Level Precision	Pixel Level Recall	Object Level Precision	Object Level Recall	Object Level F1
EfficientNet-B3	0.76	0.87	0.86	0.54	0.31	0.39
EfficientNet-B5	0.76	0.88	0.86	0.53	0.33	0.41
EfficientNet-B7	0.77	0.88	0.86	0.58	0.35	0.44

Here are the CC-BY-NC checkpoint metrics from the paper for reference:

Model	Pixel Level IoU	Pixel Level Precision	Pixel Level Recall	Object Level Precision	Object Level Recall	Object Level F1
EfficientNet-B3	0.74	-	-	-	-	0.43
EfficientNet-B5	0.75	-	-	-	-	0.46
EfficientNet-B7	0.76	0.89	0.83	0.62	0.40	0.47

Convert the checkpoints for TorchGeo compatibility

import os
import segmentation_models_pytorch as smp
import torch
import hashlib

urls = {
    "efficientnet-b3": "https://github.com/fieldsoftheworld/ftw-baselines/releases/download/v3.1/prue_efnetb3_ccby_checkpoint.ckpt",
    "efficientnet-b5": "https://github.com/fieldsoftheworld/ftw-baselines/releases/download/v3.1/prue_efnetb5_ccby_checkpoint.ckpt",
    "efficientnet-b7": "https://github.com/fieldsoftheworld/ftw-baselines/releases/download/v3.1/prue_efnetb7_ccby_checkpoint.ckpt",
}
for encoder_name, url in urls.items():
    state_dict = torch.hub.load_state_dict_from_url(url, weights_only=True, map_location="cpu")["state_dict"]
    state_dict = {k.replace("model.", ""): v for k, v in state_dict.items()}
    model = smp.Unet(encoder_name=encoder_name, in_channels=8, classes=3, encoder_weights=None)
    model.load_state_dict(state_dict)
    filename = os.path.basename(url).replace("_checkpoint.ckpt", ".pth")
    torch.save(model.state_dict(), filename)
    md5 = hashlib.md5(open(filename, "rb").read()).hexdigest()[:8]
    os.rename(filename, filename.replace(".pth", f"-{md5}.pth"))