LESSViT-Base

Pretrained LESSViT encoder for hyperspectral representation learning, exported from the LESSVIT_b2_d8_r1 checkpoint. The root of this repository contains an encoder-only Hugging Face model, with standalone implementation files and a working AutoModel interface. No project checkout is required.

• Paper
• Project website and updated manuscript
• Source repository

Intended Use and Limitations

This checkpoint was pretrained on a partial subset of EnMAP data, not the complete EnMAP corpus. It is released primarily to investigate whether explicit, efficient spatial–spectral modeling can improve representation robustness under changes in spectral configuration, including band sampling, channel count, and simulated sensor responses.

This release should not be interpreted as a fully optimized EnMAP foundation model or as evidence of universally superior downstream performance. Its training coverage, pretraining scale, and optimization settings reflect the scope of an architectural study. The reported experiments support conclusions under the evaluated protocols; they do not establish optimal performance on the full EnMAP dataset, unseen geographic distributions, or arbitrary downstream tasks.

Performance depends on the target data, preprocessing, spectral configuration, and adaptation procedure. In particular, robustness to spectral changes does not imply the highest accuracy under every fixed spectral configuration. Comparisons should account for differences in pretraining data coverage, training budget, spatial resolution, and fine-tuning protocol.

Users should independently validate this checkpoint for their intended application. Broader pretraining and task-specific adaptation may be necessary, and no guarantee is made regarding accuracy, generalization, or fitness for operational use.

Install

Reference environment: Python 3.11.

pip install torch==2.7.1 torchvision==0.22.1 transformers==4.53.3 \
  huggingface-hub==0.34.4 timm==1.0.18 numpy==2.2.6 safetensors

Choose matching PyTorch/Torchvision wheels for your hardware. This release uses custom model code, so trust_remote_code=True is required. You can pin revision to a repository commit SHA for reproducible loading.

Load the encoder directly

import torch
from transformers import AutoModel

encoder = AutoModel.from_pretrained(
    "GFM-Bench/LESSViT-Base",
    trust_remote_code=True,
).eval()

# Synthetic example: [batch, spectral channels, height, width].
x = torch.randn(1, 10, 32, 32)
wavelengths_nm = torch.linspace(450, 2300, 10)[None, :]
# Fixed full-EnMAP reference bounds for this example. Keep reference bounds
# consistent across your downstream training and evaluation configurations.
wavelengths = (wavelengths_nm - 418.24) / (2445.53 - 418.24)

with torch.no_grad():
    output = encoder(
        optical=x,
        optical_channel_wv=wavelengths,
        spatial_resolution=30,  # meters per pixel
    )

features = output.last_hidden_state
print(features.shape)                  # [1, 11, 5, 768]
global_features = features[:, 0, 0]     # [1, 768], classification
spatial_features = features[:, 0, 1:]   # [1, 4, 768], dense prediction
band_patch_features = features[:, 1:, 1:]  # [1, 10, 4, 768]

load_encoder.py contains the same runnable example:

python load_encoder.py --model-id GFM-Bench/LESSViT-Base --device cpu

To run offline after downloading a complete snapshot, pass its directory as --model-id. No network or original repository imports are needed for a local snapshot.

Inputs and outputs

• optical: float tensor [B, C, H, W]; height and width should be multiples of 16.
• optical_channel_wv: normalized wavelengths [1, C] or [B, C], matching input channel order. Use fixed reference bounds for the task, not new min/max bounds for every spectral subset. The numerical bounds above are an EnMAP example, not a universal normalization rule for every sensor.
• spatial_resolution: spatial resolution in meters per pixel.
• last_hidden_state: [B, C+1, N+1, 768], where N = (H / 16) * (W / 16). Index 0 contains aggregation tokens.

For real data, use dataset/sensor-specific intensity normalization and matching wavelength metadata. The source repository includes dataset statistics, transforms, SRF resampling, and collators. Random values in the example only verify that loading and inference work.

Attach a downstream head

# Use `encoder` and example tensors from above.
encoder.train()
head = torch.nn.Linear(encoder.config.embed_dim, 14)
features = encoder(
    optical=x, optical_channel_wv=wavelengths, spatial_resolution=30
).last_hidden_state[:, 0, 0]
logits = head(features)
loss = torch.nn.functional.cross_entropy(logits, torch.zeros(1, dtype=torch.long))
loss.backward()  # gradients through both encoder and head

For multi-label classification use binary cross entropy with logits. For dense prediction, reshape spatial_features into the patch grid and attach a spatial decoder. Task-specific heads are not included and must be trained. MAE masking and random channel dropout are disabled in this encoder interface, including when calling .train(); ordinary configured dropout remains active in training.

Architecture

Setting	Value
Encoder depth	12
Embedding dimension	768
Attention heads	12
Patch size	16
Spectral dimension per head	2
Attention rank	1
Positional encoding	Spatial–spectral RoPE
LayerScale initialization	1.0

d8 in the checkpoint name refers to the pretraining decoder, not the encoder. The encoder-only release has approximately 83.2M parameters.

Files and provenance

• Root model.safetensors: extracted encoder, with legacy attention keys remapped to the current implementation. No encoder tensors were omitted.
• Root config.json: explicit encoder architecture and Hugging Face auto-map.
• pretraining/model.safetensors: the original full checkpoint, copied byte-for-byte, including the MAE decoder. It uses the original checkpoint key layout and is not the root AutoModel format.
• pretraining/architecture.json: architectural fields from the accompanying training config. Machine-specific paths and training service settings are omitted. This is provenance metadata, not an AutoModel config.
• validation.json: export verification results.
• SHA256SUMS: file hashes.

Encoder implementation is packaged from LESSViT source commit 20fb37c10541f1a29eb2a53999afdc0734499bc3, with a small Hugging Face wrapper to preserve architecture settings across config serialization and disable MAE masking for encoder use. Loading checks compare every exported encoder tensor exactly to the supplied checkpoint after key remapping. Local AutoModel loading, forward-output equivalence, and encoder backward propagation were tested on CPU. These checks do not independently reproduce the paper's benchmark scores or establish the checkpoint's complete training-data provenance.

Citation

@misc{si2026lessvitrobusthyperspectralrepresentation,
  title={LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift},
  author={Haozhe Si and Yuxuan Wan and Yuqing Wang and Minh N. Do and Han Zhao},
  year={2026},
  eprint={2605.18541},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2605.18541}
}