README.md

metadata

license: other
tags:
  - ophthalmology
  - OCT
  - fundus
  - medical-imaging
  - diffusion
  - stable-diffusion-3
  - segmentation
  - instruction-tuning
library_name: diffusers

OCTFlow v1 — Unified Generative Instruction Foundation Model for Ophthalmic Imaging

Frozen v1 · 2026-06-21. A single SD3-medium (KL-VAE 16ch + 2B MMDiT, rectified flow) instruction-tuned (Vision-Banana style) ophthalmic foundation model: one model, change the text prompt to do generation, multi-scheme segmentation, denoising — across 9 imaging modalities. hf download MaybeRichard/OCTFlow --revision v1.

What it does (one base, switch by prompt)

• Generation (T2I) — 9 modalities: OCT B-scan, color fundus, SLO, UWF fundus, OCTA en-face, OCT en-face, FA, slit-lamp, IR-SLO.
• Instruction segmentation — OCT retinal layers (9/5/3 + arbitrary unseen counts + single-layer selection + any colors), OCT fluid (IRF/SRF/PED), OCTA FAZ & large vessels, fundus vessels & disc/cup, OCTA500 5-layer.
• Denoising — OCT speckle (generative i2i).
• Disease classification — frozen-feature linear probe (14 OCT/fundus/UWF tasks).
• Semantic-synthesis & data augmentation — mask→image generation; labeled synthetic data (shareable/privacy).

Honest results (see results/octflow_downstream_report.html + EXPERIMENT_LOG.md)

Positioning is competitive-not-SOTA per task; the genuine differentiation is unification + zero-shot instruction generalization + shareable data.

• Classification (14 tasks, linear probe, mean top1): DINOv2 0.849 ≥ RETFound 0.844 ≥ ours 0.837 > DINOv3 0.824 > MIRAGE 0.808 > VisionFM 0.796. Ours 3rd/7, never per-task #1.
• Segmentation (vs end-to-end fine-tuned FM @512, fair): ours competitive; loses most tasks to the strongest fine-tuned FM (esp. DINOv2) by 0.03–0.09, wins OCTA large-vessel, ties FIVES vessel.
• Denoising: fine-tuned FM ≥ ours on PSNR/SSIM; ours wins perceptual LPIPS (0.135 vs 0.37–0.41).
• Data augmentation (classification §5 / segmentation §6): synthetic ≈ or < classical augmentation for raw point-gain; value is in shareable/customizable data, not point-gain.
• ★ Zero-shot instruction generalization (the moat): trained only on 9/5/3-layer schemes; on unseen layer counts (8/7/6/4/2) mean mIoU 0.441 ≈ seen 0.430; cross-device zero-shot (OCTA500, never trained) binary retina IoU 0.897. A fixed-head discriminative FM / U-Net cannot do this.

Weights (weights/, optimizer state stripped, bf16)

file	role
sd3_multimodal_base_v2_step240000.pt	T2I base (generation + probe backbone)
sd3_oct_stageA_v3_step20000.pt	OCT domain-adapt init (warm-start)
sd3_vb_stageC_v3a_step30000.pt	instruction model (multi-scheme seg + zero-shot, §7)
sd3_vb_layer_100k_step30000.pt	9-layer seg anchor (v3a recipe)
sd3_vb_layer_100k_v3b_step15000.pt	9-layer seg + decoded-loss (sharper thin layers)
sd3_vb_mask2img_step12000.pt	mask→image generator (§6 semantic synthesis)
sd3_vb_denoise_100k_step4000.pt	OCT denoiser
sd3_vb_{disccup,faz,fluid,octavessel,vessel}_v3b_step4000.pt	downstream seg specialists
sd3_vb_octa500_100k_step4000.pt	OCTA500 5-layer specialist

Usage

1. hf download MaybeRichard/OCTFlow --revision v1 --local-dir octflow_v1
2. Untar octflow-raev2-code.tar.gz; install via uv sync (PyTorch 2.10+cu128). torch.load(..., weights_only=False).
3. Put a weight at pilot/path1/results/<run>/checkpoints/, set dataset paths, run the matching script in pilot/path1/scripts/foundation/ (e.g. seg_instr_eval.py, zeroshot_spectrum.py, denoise_eval.py).

Notes / limitations

• Pilot-scale; not clinically validated (external multi-center validation + reader study are future work).
• Built on gated stabilityai/stable-diffusion-3-medium-diffusers (accept its license).
• Env: base conda, torch 2.10.0+cu128, diffusers 0.37, transformers 5.3.