YoshimuraHiroto/bfm-rollout-finetuned-large

BFM Rollout Finetuned Large (BioCube, VeRA)

Rollout-finetuned version of the BioAnalyst Foundation Model (BFM) Large (702M params) on BioCube monthly climate + biodiversity data using the paper's BFMRollout pipeline (arXiv:2507.09080v2, Section 3.2).

Model Details

• Base model: BFM Large (embed_dim=512, depth=10, Swin 3D backbone, 702M params)
• PEFT: VeRA (rank=256, d_initial=0.1) — only 63.5K trainable params (0.009%)
• Finetuning data: BioCube (28 species, 0.125 deg, monthly, Europe)
• Rollout: 12-step push-forward with TBPTT-2 (gradients on last step only)
• Training: 50 epochs, AdamW (lr=5e-5, wd=5e-7), bf16-mixed
• Best val_loss: 0.627335 (epoch 36)
• Architecture: Full BFM pipeline (Encoder → Swin Backbone → Decoder), encoder/decoder frozen

Evaluation on BioCube 28 Species (48 test pairs)

Metric	Rollout FT	Pretrained	Paper
F1 (Eq.20)	0.6446	0.6450	0.9964
Sorensen (Eq.22)	0.0112	0.0110	0.31
Species MAE (norm)	0.020769	0.0210	0.0836

Key Findings

• VeRA with only 63.5K trainable parameters (0.009% of 702M) is insufficient for significant improvement over the pretrained baseline on BioCube 28 species
• The paper's results (F1=0.9964, S=0.31) likely used a much larger trainable parameter budget or different training setup (e.g., full finetune or higher-rank PEFT)
• Species predictions are mostly negative (mean=-0.019, positive_frac=3.3%), indicating the model learned to suppress species signals rather than enhance them

Usage

import torch
from safetensors.torch import load_file
from bfm_model.bfm.model import BFMRollout

# 1. Build BFMRollout with VeRA
model = BFMRollout(
    embed_dim=512, depth=10, patch_size=8,
    swin_backbone_size="large", perceiver_latents=16100,
    rollout_steps=1, finetune_mode="peft",
    use_vera=True, peft_r=256, d_initial=0.1,
    peft_mode="single", peft_steps=1,
    # ... (see finetune_rollout_large.py for full config)
)

# 2. Load pretrained base weights
state = load_file("bfm-pretrain-large.safetensors", device="cpu")
model.load_state_dict(state, strict=False)

# 3. Load finetuned VeRA weights from Lightning checkpoint
ckpt = torch.load("epoch=036-val_loss=0.627335.ckpt", map_location="cpu")
model.load_state_dict(ckpt["state_dict"], strict=False)
model.eval()

Files

• epoch=036-val_loss=0.627335.ckpt - Best Lightning checkpoint (epoch 36)
• last.ckpt - Final checkpoint (epoch 49)
• bfm_rollout_eval_results.json - Full evaluation results
• finetune_rollout_large.py - Finetuning script
• eval_rollout_large.py - Evaluation script

Citation

@article{foerster2025bfm,
  title={A Foundation Model for Forecasting Biodiversity Dynamics},
  author={Foerster et al.},
  journal={arXiv preprint arXiv:2507.09080v2},
  year={2025}
}