| # Animation transcript |
|
|
| All animated arrows are schematic. Scores are historical measurements, not simulated training curves. |
|
|
| ## 1. Read the unlabeled satellite sequence |
|
|
| **Stage:** Pretraining. 496 fold5 patches; crop labels are not loaded. |
|
|
| - **PASTISPatchDataset:** load_target=False; normalize ten bands |
| - **Sixteen fixed tiles:** 4×4 grid; each tile is32×32 |
| - **Separate crop updates:** Batch4; 1,984 optimizer steps/epoch |
| |
| Pretraining learns a representation of spectral and temporal structure. It does not directly learn the eighteen crop labels. |
| |
| Code: `pretrain.py: main() → dataset.py: crop_ij()`. |
|
|
| ## 2. Mask eligible observations |
|
|
| **Stage:** Pretraining. Historical NDVI-style frame gate, calculated on normalized bands. |
|
|
| - **Eligibility gate:** NDVI index6/2 >0.2; vegetation fraction >0.9 |
| - **Random spatial mask:** Nominal40% masking in eligible frames |
| - **Learned mask token:** Same pixel mask spans all ten spectral bands |
|
|
| Frames failing the gate remain visible. Controlled pilots found roughly3% effective masking; nominal40% is not whole-dataset coverage. |
|
|
| Code: `model.py: UTAEPrediction.forward()`. |
|
|
| ## 3. Reconstruct the hidden spectral values |
|
|
| **Stage:** Pretraining. The encoder sees corrupted inputs and predicts original ten-band values. |
|
|
| - **Spatial encoder:** Full-resolution convolutions + ASPP + SE |
| - **Temporal encoder:** Three reduced dual-attention blocks |
| - **Reconstruction head:** Linear256→10 at every pixel and time |
|
|
| Masked-value mean-squared error supplies the gradients. The auxiliary reconstruction head exists, but DEEP_SUP=False disables its extra loss. |
| |
| Code: `UTAE_ASPP.forward() → UTAEPrediction.forward()`. |
|
|
| ## 4. Save the encoder checkpoint |
|
|
| **Stage:** Pretraining. The supplied encoder artifact records epoch99. |
|
|
| - **latest.tar:** Full wrapper + optimizer/scaler/RNG state |
| - **latest.utae.tar:** Encoder weights only |
| - **Transferable representation:** Original pretrained folder retained untouched |
|
|
| We used the supplied pretrained artifact. This animation is not a claim that its original100-epoch pretraining was rerun in this session. |
|
|
| Code: `pretrain.py: main() → torch.save()`. |
|
|
| ## 5. Load your encoder; attach the supervised task |
|
|
| **Stage:** Fine-tuning. Reconstruction heads are replaced by a segmentation wrapper. |
|
|
| - **Strict encoder load:** Reuse spatial + temporal pretrained weights |
| - **Fresh supervised layers:** STA, semantic, boundary, gated refinement |
| - **Train the complete model:** Encoder and new layers both receive gradients |
|
|
| At the original supervised start, the task layers are new. Later continuation stages load the entire selected fine-tuned model, including its heads. |
|
|
| Code: `train_spatial_temporal.py: main() / --init_finetuned`. |
|
|
| ## 6. Keep the labeled sampling protocol fixed |
|
|
| **Stage:** Fine-tuning. Training fold1:76 entries /72 unique; validation fold2:76/71. |
|
|
| - **Batch of2 patches:** X:2×T×10×128×128; preserve duplicate IDs |
| - **Two fixed diagonal crops:** [0:32,0:32] and [32:64,32:64] |
| - **76 updates per epoch:** 38 batches ×2 separate crop steps |
|
|
| Training transforms operate inside these selected windows. Validation uses the same fixed windows without those random transformations. |
|
|
| Code: `dataset.py: pad_collate(), crop_ij()`. |
|
|
| ## 7. Apply the recorded training transforms |
|
|
| **Stage:** Fine-tuning. Augmentations are method-specific; sample IDs and crop windows stay fixed. |
|
|
| - **Spatial transform:** Paired flips /90-degree rotations of X and Y |
| - **Temporal transform:** Observation subsampling and frame dropout |
| - **Spectral transform:** Per-band gain/bias jitter plus noise |
|
|
| Subsampling keeps the retained observation positions. Label geometry follows the spatial transformation; validation and test are not augmented. |
|
|
| Code: `aug_dihedral(), aug_temporal_subsample(), aug_frame_dropout(), aug_spectral()`. |
|
|
| ## 8. Extract spatial features for every acquisition |
|
|
| **Stage:** Fine-tuning. The spatial encoder preserves H×W instead of downsampling the crop. |
|
|
| - **Stem10→64:** 3×3 convolution; second convolution dilation2 |
| - **ASPP64→256:** Dilations1/2/4 plus global pooling |
| - **Channel attention:** SE weighting; reshape to per-pixel sequences |
|
|
| Each pixel now has a256-dimensional feature vector at every acquisition. Spatial context and temporal modeling have separate roles. |
|
|
| Code: `AtrousSpatialEncoder.forward() / ASPP.forward()`. |
|
|
| ## 9. Model crop evolution with temporal dual attention |
|
|
| **Stage:** Fine-tuning. Index positions are encoded; actual acquisition-day gaps are not used. |
|
|
| - **Reduced KV tokens:** 256→64; group4 observations→256 |
| - **Two attention operations:** Reduced self-attention; full-T queries to reduced KV |
| - **Three residual blocks:** Eight heads; GELU + parallel depthwiseConv1D MLP |
|
|
| The winning backbone preserves its original padding behavior. Explicit padding-mask variants existed, but they are not the deployed encoder. |
|
|
| Code: `TemporalSRDABlock.forward() / TemporalSRDAEncoder.forward()`. |
|
|
| ## 10. Collapse time and predict class / boundary outputs |
|
|
| **Stage:** Fine-tuning. STA pools time; three spatial convolutions refine the pooled map. |
|
|
| - **Learned temporal pooling:** softmax scores over T → weighted feature sum |
| - **Semantic + boundary heads:** 20 class logits and1 boundary logit |
| - **Gated refinement:** Features ×(1+sigmoid(boundary)); residual class correction |
|
|
| Boundary predictions directly affect the refined class logits inside the network. They are not an external mask-cleaning stage. |
|
|
| Code: `STA.forward() → UTAEClassificationDual.forward()`. |
|
|
| ## 11. Compute the supervised objective |
|
|
| **Stage:** Fine-tuning. Two segmentation losses and auxiliary boundary supervision. |
|
|
| - **Semantic objective:** CE +0.75 Lovász |
| - **Refined objective:** CE +0.75 Lovász |
| - **Boundary objective:** 0.5 BCE; local max/min label-gradient target |
|
|
| CE/Lovász ignore GT0/19; label smoothing0.05. Boundary BCE is applied to the constructed boundary map without that foreground mask. |
|
|
| Code: `training_loss(), compute_boundary_target(), LovaszSoftmaxLoss.forward()`. |
|
|
| ## 12. Backpropagate; update encoder and task layers |
|
|
| **Stage:** Fine-tuning. AdamW parameter groups give the encoder a lower learning rate. |
|
|
| - **Gradient computation:** FP16 forward + GradScaler backward |
| - **Stabilize the update:** Unscale gradients; clip norm5 |
| - **Parameter update:** Encoder LR0.5×head LR; update EMA copy |
|
|
| Raw and EMA weights are evaluated, but the final SAM component uses the selected raw checkpoint. EMA maintenance is not proof of an EMA gain. |
|
|
| Code: `build_param_groups(), lr_factor(), ModelEMA.update()`. |
|
|
| ## 13. Continue from the low-learning-rate checkpoint |
|
|
| **Stage:** SAM refinement. Same architecture and losses; change the optimization step. |
|
|
| - **Selected low-LR model:** Load encoder AND supervised heads |
| - **Low learning rates:** Head1e-5; encoder5e-6 |
| - **SAM budget:** 20 epochs; radius0.02; batch2; fixed32 |
|
|
| The SAM run starts from a previously fine-tuned checkpoint, not directly from a bare pretrained encoder. Its optimizer and schedule start fresh. |
|
|
| Code: `main(): --init_finetuned ... --research sam`. |
|
|
| ## 14. First SAM pass: find a weight perturbation |
|
|
| **Stage:** SAM refinement. Calculate the gradient at current weights θ. |
|
|
| - **Loss atθ:** Forward → loss → backward |
| - **Normalize gradient:** g /(||g|| + epsilon) |
| - **Temporary perturbation:** θtemporary = θ +0.02 ×normalized gradient |
|
|
| This is Sharpness-Aware Minimization. No Segment Anything encoder, prompt, mask generator, or extra architecture is introduced. |
|
|
| Code: `train_spatial_temporal.py: main(), research==sam branch`. |
|
|
| ## 15. Second SAM pass: restore and update |
|
|
| **Stage:** SAM refinement. Compute a new gradient at the temporarily perturbed weights. |
|
|
| - **Second loss/backward:** Same crop; BatchNorm held in eval mode |
| - **Restore originalθ:** Subtract stored perturbations |
| - **AdamW update:** Apply second-pass gradient; update scaler/EMA |
|
|
| Select raw epoch11 using validation. Once saved, the network needs only its ordinary forward pass at inference. |
|
|
| Code: `research==sam: offsets → second_loss → restore → scaler.step()`. |
|
|
| ## 16. Freeze the SAM encoder and collect earlier features |
|
|
| **Stage:** Feature clusters. Reuse its first temporal block, rather than training another encoder. |
|
|
| - **Forward hook:** DepthCollector captures block1 output |
| - **Valid-time statistics:** Mean256 +standard deviation256 =vector512 |
| - **Training normalization:** Fit mean/std on training features; then L2 normalize |
|
|
| The readout shares the network forward pass. Hooks summarize real features; the diagram does not visualize measured feature activations. |
|
|
| Code: `DepthCollector.begin(), hook(), end(); train_normalizer()`. |
|
|
| ## 17. Fit three feature-cluster centres per crop class |
|
|
| **Stage:** Feature clusters. Use labeled training pixels only; no validation/test labels fit centres. |
|
|
| - **Balanced memory:** 32 pixels per supporting training patch/class |
| - **KMeans fitting:** 18 classes ×3 centres; seed3407; n_init3 |
| - **Similarity classifier:** Similarity/0.1 → logsumexp over3 → softmax18 |
| |
| Repeated patch/crop occurrences are deduplicated for the memory bank only. The supervised dataset list itself retains duplicate entries. |
| |
| Code: `make_prototypes() / probability()`. |
|
|
| ## 18. Train the original independent temporal specialist |
|
|
| **Stage:** Temporal CNNs. A small model of each pixel’s spectral evolution, trained from scratch. |
|
|
| - **Input projection:** T×10 → T×64; valid-time mask |
| - **Four Conv1D blocks:** Kernel3; depthwise; dilation1/2/4/8 |
| - **Mean +max classifier:** Vector128 →64 →18 crop classes |
|
|
| 100 epochs; AdamW1e-3, WD0.01, CE smoothing0.05, warmup3/cosine. Combine25% CNN probabilities with75% SAM/cluster blend. |
|
|
| Code: `TemporalCNN.forward(); historical original CNN training`. |
|
|
| ## 19. Add the new plain-CNN control |
|
|
| **Stage:** Temporal CNNs. A separate training run with the same architecture and seed value. |
|
|
| - **Matched control:** Same data, fixed32 crops,100-epoch budget |
| - **Distinct training trajectory:** New trainer resets RNG after model construction |
| - **Validation-selected blend:** 75% reference +25% new CNN probabilities |
|
|
| The CNN is weaker alone but changes useful class rankings. A matched attention alternative replaces mean pooling and adds65 parameters. |
|
|
| Code: `train_temporal_cnn.py: train(); AttentionCNN.forward()`. |
|
|
| ## 20. Run the complete system on full128 test patches |
|
|
| **Stage:** Final evaluation. 482 fold4 patches; no TTA or external mask post-processing. |
|
|
| - **Four probability sources:** SAM /shared-feature clusters /old CNN /new CNN |
| - **Recommended weights:** 28.125% /28.125% /18.75% /25% |
| - **Final argmax +metrics:** Full20×20 matrix; score GT classes1–18 |
|
|
| Recommended: validation52.5640%, test49.4555%. Attention alternative: validation51.8368%, highest observed test49.5298%; reported separately. |
|
|
| Code: `evaluate.py: load_models(), main(); metrics_cm()`. |
