# Animation transcript

All animated arrows are schematic. Scores are historical measurements, not simulated training curves.

## 1. Read the unlabeled satellite sequence

**Stage:** Pretraining. 496 fold5 patches; crop labels are not loaded.

- **PASTISPatchDataset:** load_target=False; normalize ten bands
- **Sixteen fixed tiles:** 4×4 grid; each tile is32×32
- **Separate crop updates:** Batch4; 1,984 optimizer steps/epoch

Pretraining learns a representation of spectral and temporal structure. It does not directly learn the eighteen crop labels.

Code: `pretrain.py: main() → dataset.py: crop_ij()`.

## 2. Mask eligible observations

**Stage:** Pretraining. Historical NDVI-style frame gate, calculated on normalized bands.

- **Eligibility gate:** NDVI index6/2 >0.2; vegetation fraction >0.9
- **Random spatial mask:** Nominal40% masking in eligible frames
- **Learned mask token:** Same pixel mask spans all ten spectral bands

Frames failing the gate remain visible. Controlled pilots found roughly3% effective masking; nominal40% is not whole-dataset coverage.

Code: `model.py: UTAEPrediction.forward()`.

## 3. Reconstruct the hidden spectral values

**Stage:** Pretraining. The encoder sees corrupted inputs and predicts original ten-band values.

- **Spatial encoder:** Full-resolution convolutions + ASPP + SE
- **Temporal encoder:** Three reduced dual-attention blocks
- **Reconstruction head:** Linear256→10 at every pixel and time

Masked-value mean-squared error supplies the gradients. The auxiliary reconstruction head exists, but DEEP_SUP=False disables its extra loss.

Code: `UTAE_ASPP.forward() → UTAEPrediction.forward()`.

## 4. Save the encoder checkpoint

**Stage:** Pretraining. The supplied encoder artifact records epoch99.

- **latest.tar:** Full wrapper + optimizer/scaler/RNG state
- **latest.utae.tar:** Encoder weights only
- **Transferable representation:** Original pretrained folder retained untouched

We used the supplied pretrained artifact. This animation is not a claim that its original100-epoch pretraining was rerun in this session.

Code: `pretrain.py: main() → torch.save()`.

## 5. Load your encoder; attach the supervised task

**Stage:** Fine-tuning. Reconstruction heads are replaced by a segmentation wrapper.

- **Strict encoder load:** Reuse spatial + temporal pretrained weights
- **Fresh supervised layers:** STA, semantic, boundary, gated refinement
- **Train the complete model:** Encoder and new layers both receive gradients

At the original supervised start, the task layers are new. Later continuation stages load the entire selected fine-tuned model, including its heads.

Code: `train_spatial_temporal.py: main() / --init_finetuned`.

## 6. Keep the labeled sampling protocol fixed

**Stage:** Fine-tuning. Training fold1:76 entries /72 unique; validation fold2:76/71.

- **Batch of2 patches:** X:2×T×10×128×128; preserve duplicate IDs
- **Two fixed diagonal crops:** [0:32,0:32] and [32:64,32:64]
- **76 updates per epoch:** 38 batches ×2 separate crop steps

Training transforms operate inside these selected windows. Validation uses the same fixed windows without those random transformations.

Code: `dataset.py: pad_collate(), crop_ij()`.

## 7. Apply the recorded training transforms

**Stage:** Fine-tuning. Augmentations are method-specific; sample IDs and crop windows stay fixed.

- **Spatial transform:** Paired flips /90-degree rotations of X and Y
- **Temporal transform:** Observation subsampling and frame dropout
- **Spectral transform:** Per-band gain/bias jitter plus noise

Subsampling keeps the retained observation positions. Label geometry follows the spatial transformation; validation and test are not augmented.

Code: `aug_dihedral(), aug_temporal_subsample(), aug_frame_dropout(), aug_spectral()`.

## 8. Extract spatial features for every acquisition

**Stage:** Fine-tuning. The spatial encoder preserves H×W instead of downsampling the crop.

- **Stem10→64:** 3×3 convolution; second convolution dilation2
- **ASPP64→256:** Dilations1/2/4 plus global pooling
- **Channel attention:** SE weighting; reshape to per-pixel sequences

Each pixel now has a256-dimensional feature vector at every acquisition. Spatial context and temporal modeling have separate roles.

Code: `AtrousSpatialEncoder.forward() / ASPP.forward()`.

## 9. Model crop evolution with temporal dual attention

**Stage:** Fine-tuning. Index positions are encoded; actual acquisition-day gaps are not used.

- **Reduced KV tokens:** 256→64; group4 observations→256
- **Two attention operations:** Reduced self-attention; full-T queries to reduced KV
- **Three residual blocks:** Eight heads; GELU + parallel depthwiseConv1D MLP

The winning backbone preserves its original padding behavior. Explicit padding-mask variants existed, but they are not the deployed encoder.

Code: `TemporalSRDABlock.forward() / TemporalSRDAEncoder.forward()`.

## 10. Collapse time and predict class / boundary outputs

**Stage:** Fine-tuning. STA pools time; three spatial convolutions refine the pooled map.

- **Learned temporal pooling:** softmax scores over T → weighted feature sum
- **Semantic + boundary heads:** 20 class logits and1 boundary logit
- **Gated refinement:** Features ×(1+sigmoid(boundary)); residual class correction

Boundary predictions directly affect the refined class logits inside the network. They are not an external mask-cleaning stage.

Code: `STA.forward() → UTAEClassificationDual.forward()`.

## 11. Compute the supervised objective

**Stage:** Fine-tuning. Two segmentation losses and auxiliary boundary supervision.

- **Semantic objective:** CE +0.75 Lovász
- **Refined objective:** CE +0.75 Lovász
- **Boundary objective:** 0.5 BCE; local max/min label-gradient target

CE/Lovász ignore GT0/19; label smoothing0.05. Boundary BCE is applied to the constructed boundary map without that foreground mask.

Code: `training_loss(), compute_boundary_target(), LovaszSoftmaxLoss.forward()`.

## 12. Backpropagate; update encoder and task layers

**Stage:** Fine-tuning. AdamW parameter groups give the encoder a lower learning rate.

- **Gradient computation:** FP16 forward + GradScaler backward
- **Stabilize the update:** Unscale gradients; clip norm5
- **Parameter update:** Encoder LR0.5×head LR; update EMA copy

Raw and EMA weights are evaluated, but the final SAM component uses the selected raw checkpoint. EMA maintenance is not proof of an EMA gain.

Code: `build_param_groups(), lr_factor(), ModelEMA.update()`.

## 13. Continue from the low-learning-rate checkpoint

**Stage:** SAM refinement. Same architecture and losses; change the optimization step.

- **Selected low-LR model:** Load encoder AND supervised heads
- **Low learning rates:** Head1e-5; encoder5e-6
- **SAM budget:** 20 epochs; radius0.02; batch2; fixed32

The SAM run starts from a previously fine-tuned checkpoint, not directly from a bare pretrained encoder. Its optimizer and schedule start fresh.

Code: `main(): --init_finetuned ... --research sam`.

## 14. First SAM pass: find a weight perturbation

**Stage:** SAM refinement. Calculate the gradient at current weights θ.

- **Loss atθ:** Forward → loss → backward
- **Normalize gradient:** g /(||g|| + epsilon)
- **Temporary perturbation:** θtemporary = θ +0.02 ×normalized gradient

This is Sharpness-Aware Minimization. No Segment Anything encoder, prompt, mask generator, or extra architecture is introduced.

Code: `train_spatial_temporal.py: main(), research==sam branch`.

## 15. Second SAM pass: restore and update

**Stage:** SAM refinement. Compute a new gradient at the temporarily perturbed weights.

- **Second loss/backward:** Same crop; BatchNorm held in eval mode
- **Restore originalθ:** Subtract stored perturbations
- **AdamW update:** Apply second-pass gradient; update scaler/EMA

Select raw epoch11 using validation. Once saved, the network needs only its ordinary forward pass at inference.

Code: `research==sam: offsets → second_loss → restore → scaler.step()`.

## 16. Freeze the SAM encoder and collect earlier features

**Stage:** Feature clusters. Reuse its first temporal block, rather than training another encoder.

- **Forward hook:** DepthCollector captures block1 output
- **Valid-time statistics:** Mean256 +standard deviation256 =vector512
- **Training normalization:** Fit mean/std on training features; then L2 normalize

The readout shares the network forward pass. Hooks summarize real features; the diagram does not visualize measured feature activations.

Code: `DepthCollector.begin(), hook(), end(); train_normalizer()`.

## 17. Fit three feature-cluster centres per crop class

**Stage:** Feature clusters. Use labeled training pixels only; no validation/test labels fit centres.

- **Balanced memory:** 32 pixels per supporting training patch/class
- **KMeans fitting:** 18 classes ×3 centres; seed3407; n_init3
- **Similarity classifier:** Similarity/0.1 → logsumexp over3 → softmax18

Repeated patch/crop occurrences are deduplicated for the memory bank only. The supervised dataset list itself retains duplicate entries.

Code: `make_prototypes() / probability()`.

## 18. Train the original independent temporal specialist

**Stage:** Temporal CNNs. A small model of each pixel’s spectral evolution, trained from scratch.

- **Input projection:** T×10 → T×64; valid-time mask
- **Four Conv1D blocks:** Kernel3; depthwise; dilation1/2/4/8
- **Mean +max classifier:** Vector128 →64 →18 crop classes

100 epochs; AdamW1e-3, WD0.01, CE smoothing0.05, warmup3/cosine. Combine25% CNN probabilities with75% SAM/cluster blend.

Code: `TemporalCNN.forward(); historical original CNN training`.

## 19. Add the new plain-CNN control

**Stage:** Temporal CNNs. A separate training run with the same architecture and seed value.

- **Matched control:** Same data, fixed32 crops,100-epoch budget
- **Distinct training trajectory:** New trainer resets RNG after model construction
- **Validation-selected blend:** 75% reference +25% new CNN probabilities

The CNN is weaker alone but changes useful class rankings. A matched attention alternative replaces mean pooling and adds65 parameters.

Code: `train_temporal_cnn.py: train(); AttentionCNN.forward()`.

## 20. Run the complete system on full128 test patches

**Stage:** Final evaluation. 482 fold4 patches; no TTA or external mask post-processing.

- **Four probability sources:** SAM /shared-feature clusters /old CNN /new CNN
- **Recommended weights:** 28.125% /28.125% /18.75% /25%
- **Final argmax +metrics:** Full20×20 matrix; score GT classes1–18

Recommended: validation52.5640%, test49.4555%. Attention alternative: validation51.8368%, highest observed test49.5298%; reported separately.

Code: `evaluate.py: load_models(), main(); metrics_cm()`.