visuals/finetuning_walkthrough_transcript.md

Animation transcript

All animated arrows are schematic. Scores are historical measurements, not simulated training curves.

1. Read the unlabeled satellite sequence

Stage: Pretraining. 496 fold5 patches; crop labels are not loaded.

• PASTISPatchDataset: load_target=False; normalize ten bands
• Sixteen fixed tiles: 4×4 grid; each tile is32×32
• Separate crop updates: Batch4; 1,984 optimizer steps/epoch

Pretraining learns a representation of spectral and temporal structure. It does not directly learn the eighteen crop labels.

Code: pretrain.py: main() → dataset.py: crop_ij().

2. Mask eligible observations

Stage: Pretraining. Historical NDVI-style frame gate, calculated on normalized bands.

• Eligibility gate: NDVI index6/2 >0.2; vegetation fraction >0.9
• Random spatial mask: Nominal40% masking in eligible frames
• Learned mask token: Same pixel mask spans all ten spectral bands

Frames failing the gate remain visible. Controlled pilots found roughly3% effective masking; nominal40% is not whole-dataset coverage.

Code: model.py: UTAEPrediction.forward().

3. Reconstruct the hidden spectral values

Stage: Pretraining. The encoder sees corrupted inputs and predicts original ten-band values.

• Spatial encoder: Full-resolution convolutions + ASPP + SE
• Temporal encoder: Three reduced dual-attention blocks
• Reconstruction head: Linear256→10 at every pixel and time

Masked-value mean-squared error supplies the gradients. The auxiliary reconstruction head exists, but DEEP_SUP=False disables its extra loss.

Code: UTAE_ASPP.forward() → UTAEPrediction.forward().

4. Save the encoder checkpoint

Stage: Pretraining. The supplied encoder artifact records epoch99.

• latest.tar: Full wrapper + optimizer/scaler/RNG state
• latest.utae.tar: Encoder weights only
• Transferable representation: Original pretrained folder retained untouched

We used the supplied pretrained artifact. This animation is not a claim that its original100-epoch pretraining was rerun in this session.

Code: pretrain.py: main() → torch.save().

5. Load your encoder; attach the supervised task

Stage: Fine-tuning. Reconstruction heads are replaced by a segmentation wrapper.

• Strict encoder load: Reuse spatial + temporal pretrained weights
• Fresh supervised layers: STA, semantic, boundary, gated refinement
• Train the complete model: Encoder and new layers both receive gradients

At the original supervised start, the task layers are new. Later continuation stages load the entire selected fine-tuned model, including its heads.

Code: train_spatial_temporal.py: main() / --init_finetuned.

6. Keep the labeled sampling protocol fixed

Stage: Fine-tuning. Training fold1:76 entries /72 unique; validation fold2:76/71.

• Batch of2 patches: X:2×T×10×128×128; preserve duplicate IDs
• Two fixed diagonal crops: [0:32,0:32] and [32:64,32:64]
• 76 updates per epoch: 38 batches ×2 separate crop steps

Training transforms operate inside these selected windows. Validation uses the same fixed windows without those random transformations.

Code: dataset.py: pad_collate(), crop_ij().

7. Apply the recorded training transforms

Stage: Fine-tuning. Augmentations are method-specific; sample IDs and crop windows stay fixed.

• Spatial transform: Paired flips /90-degree rotations of X and Y
• Temporal transform: Observation subsampling and frame dropout
• Spectral transform: Per-band gain/bias jitter plus noise

Subsampling keeps the retained observation positions. Label geometry follows the spatial transformation; validation and test are not augmented.

Code: aug_dihedral(), aug_temporal_subsample(), aug_frame_dropout(), aug_spectral().

8. Extract spatial features for every acquisition

Stage: Fine-tuning. The spatial encoder preserves H×W instead of downsampling the crop.

• Stem10→64: 3×3 convolution; second convolution dilation2
• ASPP64→256: Dilations1/2/4 plus global pooling
• Channel attention: SE weighting; reshape to per-pixel sequences

Each pixel now has a256-dimensional feature vector at every acquisition. Spatial context and temporal modeling have separate roles.

Code: AtrousSpatialEncoder.forward() / ASPP.forward().

9. Model crop evolution with temporal dual attention

Stage: Fine-tuning. Index positions are encoded; actual acquisition-day gaps are not used.

• Reduced KV tokens: 256→64; group4 observations→256
• Two attention operations: Reduced self-attention; full-T queries to reduced KV
• Three residual blocks: Eight heads; GELU + parallel depthwiseConv1D MLP

The winning backbone preserves its original padding behavior. Explicit padding-mask variants existed, but they are not the deployed encoder.

Code: TemporalSRDABlock.forward() / TemporalSRDAEncoder.forward().

10. Collapse time and predict class / boundary outputs

Stage: Fine-tuning. STA pools time; three spatial convolutions refine the pooled map.

• Learned temporal pooling: softmax scores over T → weighted feature sum
• Semantic + boundary heads: 20 class logits and1 boundary logit
• Gated refinement: Features ×(1+sigmoid(boundary)); residual class correction

Boundary predictions directly affect the refined class logits inside the network. They are not an external mask-cleaning stage.

Code: STA.forward() → UTAEClassificationDual.forward().

11. Compute the supervised objective

Stage: Fine-tuning. Two segmentation losses and auxiliary boundary supervision.

• Semantic objective: CE +0.75 Lovász
• Refined objective: CE +0.75 Lovász
• Boundary objective: 0.5 BCE; local max/min label-gradient target

CE/Lovász ignore GT0/19; label smoothing0.05. Boundary BCE is applied to the constructed boundary map without that foreground mask.

Code: training_loss(), compute_boundary_target(), LovaszSoftmaxLoss.forward().

12. Backpropagate; update encoder and task layers

Stage: Fine-tuning. AdamW parameter groups give the encoder a lower learning rate.

• Gradient computation: FP16 forward + GradScaler backward
• Stabilize the update: Unscale gradients; clip norm5
• Parameter update: Encoder LR0.5×head LR; update EMA copy

Raw and EMA weights are evaluated, but the final SAM component uses the selected raw checkpoint. EMA maintenance is not proof of an EMA gain.

Code: build_param_groups(), lr_factor(), ModelEMA.update().

13. Continue from the low-learning-rate checkpoint

Stage: SAM refinement. Same architecture and losses; change the optimization step.

• Selected low-LR model: Load encoder AND supervised heads
• Low learning rates: Head1e-5; encoder5e-6
• SAM budget: 20 epochs; radius0.02; batch2; fixed32

The SAM run starts from a previously fine-tuned checkpoint, not directly from a bare pretrained encoder. Its optimizer and schedule start fresh.

Code: main(): --init_finetuned ... --research sam.

14. First SAM pass: find a weight perturbation

Stage: SAM refinement. Calculate the gradient at current weights θ.

• Loss atθ: Forward → loss → backward
• Normalize gradient: g /(||g|| + epsilon)
• Temporary perturbation: θtemporary = θ +0.02 ×normalized gradient

This is Sharpness-Aware Minimization. No Segment Anything encoder, prompt, mask generator, or extra architecture is introduced.

Code: train_spatial_temporal.py: main(), research==sam branch.

15. Second SAM pass: restore and update

Stage: SAM refinement. Compute a new gradient at the temporarily perturbed weights.

• Second loss/backward: Same crop; BatchNorm held in eval mode
• Restore originalθ: Subtract stored perturbations
• AdamW update: Apply second-pass gradient; update scaler/EMA

Select raw epoch11 using validation. Once saved, the network needs only its ordinary forward pass at inference.

Code: research==sam: offsets → second_loss → restore → scaler.step().

16. Freeze the SAM encoder and collect earlier features

Stage: Feature clusters. Reuse its first temporal block, rather than training another encoder.

• Forward hook: DepthCollector captures block1 output
• Valid-time statistics: Mean256 +standard deviation256 =vector512
• Training normalization: Fit mean/std on training features; then L2 normalize

The readout shares the network forward pass. Hooks summarize real features; the diagram does not visualize measured feature activations.

Code: DepthCollector.begin(), hook(), end(); train_normalizer().

17. Fit three feature-cluster centres per crop class

Stage: Feature clusters. Use labeled training pixels only; no validation/test labels fit centres.

• Balanced memory: 32 pixels per supporting training patch/class
• KMeans fitting: 18 classes ×3 centres; seed3407; n_init3
• Similarity classifier: Similarity/0.1 → logsumexp over3 → softmax18

Repeated patch/crop occurrences are deduplicated for the memory bank only. The supervised dataset list itself retains duplicate entries.

Code: make_prototypes() / probability().

18. Train the original independent temporal specialist

Stage: Temporal CNNs. A small model of each pixel’s spectral evolution, trained from scratch.

• Input projection: T×10 → T×64; valid-time mask
• Four Conv1D blocks: Kernel3; depthwise; dilation1/2/4/8
• Mean +max classifier: Vector128 →64 →18 crop classes

100 epochs; AdamW1e-3, WD0.01, CE smoothing0.05, warmup3/cosine. Combine25% CNN probabilities with75% SAM/cluster blend.

Code: TemporalCNN.forward(); historical original CNN training.

19. Add the new plain-CNN control

Stage: Temporal CNNs. A separate training run with the same architecture and seed value.

• Matched control: Same data, fixed32 crops,100-epoch budget
• Distinct training trajectory: New trainer resets RNG after model construction
• Validation-selected blend: 75% reference +25% new CNN probabilities

The CNN is weaker alone but changes useful class rankings. A matched attention alternative replaces mean pooling and adds65 parameters.

Code: train_temporal_cnn.py: train(); AttentionCNN.forward().

20. Run the complete system on full128 test patches

Stage: Final evaluation. 482 fold4 patches; no TTA or external mask post-processing.

• Four probability sources: SAM /shared-feature clusters /old CNN /new CNN
• Recommended weights: 28.125% /28.125% /18.75% /25%
• Final argmax +metrics: Full20×20 matrix; score GT classes1–18

Recommended: validation52.5640%, test49.4555%. Attention alternative: validation51.8368%, highest observed test49.5298%; reported separately.

Code: evaluate.py: load_models(), main(); metrics_cm().