README.md

---
library_name: pytorch
pipeline_tag: feature-extraction
base_model: nvidia/mit-b2
license: other
tags:
  - remote-sensing
  - multispectral
  - time-series
  - self-supervised-learning
  - forest-phenology
  - tree-crowns
  - uav-imagery
---

# PhenoEmbed MiT-B2

PhenoEmbed is a self-supervised temporal feature extractor for individual tree
crowns observed in multispectral UAV image time series. It converts an aligned
sequence of crown-centered multispectral crops into one L2-normalized,
256-dimensional vector summarizing seasonal crown appearance.

The model was developed to capture phenological changes such as leaf emergence,
canopy closure, senescence, and leaf-off conditions. It is intended as a
representation model for downstream tree-level Earth-observation tasks rather
than as a species classifier or segmentation model.



![phenoembed](https://cdn-uploads.huggingface.co/production/uploads/678d3a445e6af698f2d2ce30/SQA7OPZL6AO9rDCQtOP8m.png)


## Model Description

- **Architecture:** SegFormer MiT-B2 spatial encoder with temporal Transformer
- **Base model:** [`nvidia/mit-b2`](https://huggingface.co/nvidia/mit-b2)
- **Input bands:** red, green, red-edge, and near-infrared
- **Temporal observations:** 18 acquisition dates
- **Crop extent:** 16 m × 16 m per crown
- **Stored crop resolution:** 288 × 288 pixels
- **Backbone input resolution:** 224 × 224 pixels
- **Output:** one L2-normalized 256-dimensional embedding per crown
- **Training method:** self-supervised temporal contrastive learning and masked temporal reconstruction
- **Spatial backbone:** frozen during training
- **Trainable components:** multispectral adapter, temporal Transformer, projection head, and reconstruction head

A trainable 1 × 1 convolution maps the four multispectral bands to the
three-channel input expected by the ImageNet-pretrained MiT-B2 backbone. It is
initialized with red and green passed to their corresponding channels and the
third channel initialized from the mean of red-edge and NIR. All adapter
weights remain trainable.

Per-date spatial features are combined with normalized seasonal time features
using a two-layer, four-head temporal Transformer.

## Training Data

The model was trained on
[HeideBench](https://doi.org/10.1594/PANGAEA.993969), a multispectral UAV
time-series dataset covering a forest patch in Dölauer Heide, Halle (Saale),
Germany.

The training corpus contains:

- 18 UAV orthomosaics acquired between 6 March and 5 November 2025
- 5,885 crop-safe individual tree crowns
- 105,930 crown-date crop instances
- 5,297 training crowns and 588 validation crowns
- Canonical band order: R, G, RE, NIR
- Average source ground sampling distance: 5.53 cm per pixel

Crown polygons were used as fixed object anchors for extracting aligned crops
of the same tree through time.

## Input Preprocessing

Input reflectance values must be arranged in canonical order:

```text
R, G, RE, NIR
```

The preprocessing used during training was:

1. Divide reflectance values by 10,000.
2. Clip values to the range `[0, 4]`.
3. Normalize each band using the following statistics:

| Band | Mean | Standard deviation |
|---|---:|---:|
| R | 0.06105786 | 0.05220907 |
| G | 0.07939660 | 0.05130135 |
| RE | 0.31415916 | 0.20160384 |
| NIR | 0.67597259 | 0.40058157 |

Invalid pixels are excluded using the accompanying alpha mask.

Each acquisition date also receives a normalized seasonal coordinate:

```text
(day - first_day) / (last_day - first_day)
```

Inputs from other sensors or sites should be calibrated to comparable
reflectance units. Reusing the HeideBench normalization statistics outside the
training domain may not be appropriate.

## Training Configuration

- Batch size: 2
- Completed optimization steps: 50,000
- Optimizer: AdamW
- Learning rate: 5 × 10⁻⁵
- Weight decay: 10⁻⁴
- Temporal masking probability: 0.3
- Contrastive temperature: 0.2
- Contrastive-loss weight: 1.0
- Reconstruction-loss weight: 1.0
- Precision: mixed 16-bit
- Checkpoint selection: minimum validation objective

A separate batch-size-16 sensitivity run was also evaluated. It did not improve
the intrinsic embedding diagnostics under its training schedule, but it used a
different learning rate and stopping configuration and should not be interpreted
as a controlled batch-size ablation.

## Usage

PhenoEmbed uses a custom PyTorch Lightning architecture and cannot be loaded
directly with `transformers.AutoModel`.

Clone and install the PhenoEmbed repository, prepare a compatible crop manifest,
and run:

```bash
PYTHONPATH=src python -m phenoembed.inference.export_embeddings \
  --checkpoint-path phenoembed-mitb2-full.ckpt \
  --data-config configs/dataloader_full.toml \
  --output-path outputs/crown_embeddings.csv \
  --npz-path outputs/crown_embeddings.npz \
  --device cuda
```

The CSV contains the crown identifier, acquisition-date sequence, and 256
embedding dimensions. The optional NPZ output contains `crown_id`,
`date_sequence`, and `embedding` arrays.

## Evaluation

Intrinsic evaluation on 5,885 HeideBench crowns produced:

| Diagnostic | Result |
|---|---:|
| Variance explained by PC1 and PC2 | 25.1% |
| Variance explained by the first 8 PCs | 71.8% |
| Median top-1 cosine similarity | 0.946 |
| Median cosine similarity among top-10 neighbors | 0.902 |
| NDVI-amplitude linear probe, five-fold CV R² | 0.525 |
| NDRE-amplitude linear probe, five-fold CV R² | 0.414 |

The linear-probe results show that the embeddings retain measurable information
about seasonal vegetation change. PCA and nearest-neighbor similarity are
intrinsic representation diagnostics, not downstream accuracy measurements.

## Intended Uses

PhenoEmbed is intended for:

- Crown-level temporal representation extraction
- Forest phenology analysis
- Similarity search and crown retrieval
- Phenology-aware feature generation
- Research on seasonally robust tree-level models
- Future integration with crown segmentation or classification systems

## Limitations

- The model was trained on one forest site, one year, and one UAV sensor.
- Generalization across sites, years, sensors, and spatial resolutions has not
  yet been established.
- The model expects aligned observations of the same annotated crown through time.
- The selected model uses only a small number of in-batch contrastive negatives.
- The MiT-B2 backbone is frozen and receives four-band information through a
  learned four-to-three-channel adapter.
- The reconstruction objective predicts per-date band means rather than detailed
  spatial structure.
- Current evaluation is intrinsic. Improved downstream crown segmentation under
  seasonal shift has not yet been demonstrated.
- Embedding similarity must not be interpreted as species identity, ecological
  equivalence, health status, or segmentation accuracy without independent
  validation.

## Citation

```bibtex
@inproceedings{khan2026phenoembed,
  title     = {PhenoEmbed: Self-Supervised Multispectral UAV Time-Series
               Embeddings for Individual Tree Crown Phenology},
  author    = {Khan, Taimur},
  year      = {2026},
  note      = {Resilience and AI Workshop at Informatik Festival 2026}
}
```

Please also cite the training dataset:

```bibtex
@dataset{khan2026heidebench,
  author    = {Khan, Taimur},
  title     = {HeideBench: A Multispectral UAV Time-Series Benchmark for
               Forest Crown Phenology in Dölauer Heide},
  publisher = {PANGAEA},
  year      = {2026},
  doi       = {10.1594/PANGAEA.993969}
}
```

## License

The release license for the PhenoEmbed weights must be stated here before
publication. Use of the model must also comply with the terms of the pretrained
MiT-B2 model and the HeideBench dataset.