FunAILab/Open-Vision-Banana

Open Vision Banana: A Fully Open Recipe for Generative Visual Perception

Outputs of the zero-shot 9B model on an in-the-wild photo, with the summarised prompt below each output.

📦 Checkpoints

• final_models/{4B,9B}/{zeroshot,nonzeroshot}/: the fine-tuned transformers for both base sizes. The zero-shot models are trained without the training splits of the evaluation benchmarks, and the non-zero-shot models include them. Each folder holds the intermediate checkpoint-<step> exports and the weight-averaged checkpoint used in the paper, each as a transformer/ subfolder.
• ablation_models/: one checkpoint per ablation setting of the paper, and the LoRA adapters described at the bottom of this page.

🚀 Quickstart

Install PyTorch with CUDA support first, then:

pip install "diffusers>=0.37.0" transformers accelerate safetensors pillow

The fine-tuned transformer is swapped into the FLUX.2 [klein] Base 9B pipeline. One model handles every task and the prompt selects it. This example runs:

import torch
from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel
from diffusers.utils import load_image
from PIL import Image

device = "cuda"
dtype = torch.bfloat16

transformer = Flux2Transformer2DModel.from_pretrained(
    "FunAILab/Open-Vision-Banana",
    subfolder="final_models/9B/nonzeroshot/checkpoint-avg82500-85000-87500-92500-97500_uniform/transformer",
    torch_dtype=dtype,
)
pipe = Flux2KleinPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-base-9B", transformer=transformer, torch_dtype=dtype
)
pipe.enable_model_cpu_offload()  # save some VRAM by offloading the model to CPU

image = load_image("street.jpg")
prompt = (
    "[TASK: SEMANTIC_SEGMENTATION] Generate a semantic segmentation visualization of "
    'the input image, using this color mapping: {"road": [8,4,249], "sidewalk": [255,8,232], '
    '"vegetation": [10,255,2], "sky": [2,204,254], "person": [255,0,13], "car": [255,249,23]} '
    "Color each pixel of a listed class with its color and leave everything else black. "
    "Output only the segmentation visualization image."
)

# The pipeline runs at the input's aspect ratio, capped at 1 megapixel.
output = pipe(
    prompt=prompt,
    image=image,
    guidance_scale=3.5,
    num_inference_steps=28,
    generator=torch.Generator(device=device).manual_seed(0),
).images[0]
# Back to the input size: NEAREST keeps mask colours exact; use Image.BILINEAR for depth and normals.
output.resize(image.size, Image.NEAREST).save("output.png")

🎯 Quickstart prompts for the other tasks

Replace prompt above with one of these.

prompt = (
    "[TASK: REFERRING_SEGMENTATION] Segment the region that best matches this "
    'description: "table in the room". Paint the matching region red [220, 30, 30] and '
    "leave everything else black [0, 0, 0]. Output only the segmentation mask image."
)

prompt = (
    "[TASK: INSTANCE_SEGMENTATION] This image shows segmentation masks for car from the input image. "
    "The background is set to black color [0, 0, 0]. Each car instance is represented by a solid mask, "
    "and a different color is used for each mask. Instances are numbered by where their mask first "
    "appears in a left-to-right scan of the image (ties broken top to bottom). The 1st car is blue, "
    "the 2nd is lime, the 3rd is red, the 4th is sky blue, the 5th is dark green, the 6th is pink, "
    "the 7th is apricot, the 8th is mint, the 9th is wine, the 10th is magenta, the 11th is navy, "
    "the 12th is green, the 13th is yellow, the 14th is brown, the 15th is white, the 16th is olive, "
    "the 17th is lavender, the 18th is teal, the 19th is orange, the 20th is hot pink, the 21st is plum, "
    "the 22nd is gray, the 23rd is purple, the 24th is beige, the 25th is steel blue, the 26th is cyan, "
    "the 27th is royal blue, the 28th is sea green, the 29th is rust, the 30th is mustard, "
    "the 31st is mauve, the 32nd is maroon."
)

prompt = (
    "Estimate a dense relative inverse-depth map for the input image. Produce exactly one output value "
    "for every input pixel at the same pixel location, with closer points brighter and farther points "
    "darker. Encode the result as neutral grayscale (R=G=B) using one consistent mapping over the "
    "complete image. Preserve object silhouettes, thin structures, occlusion boundaries, smooth surfaces, "
    "and small depth changes. Do not alter the scene or add texture, shading, color, text, labels, "
    "legends, borders, or objects; return only the registered depth map."
)

prompt = (
    "Create a dense camera-space unit surface-normal map registered to the input. Use +x image-right, "
    "+y image-up, +z toward the camera and encode (x,y,z) as R=(1-x)/2, G=(1+y)/2, B=(1+z)/2. "
    "Preserve planes, curved surfaces, thin structures, and sharp orientation boundaries; "
    "return only the RGB normal map."
)

prompt = (
    "[TASK: IMAGE_EDITING] Change the beach and ocean environment in the picture to a snowy mountain."
)

Call the pipeline without image and give it a size:

prompt = "an empty tram at a cobblestone stop on a rainy night"
output = pipe(
    prompt=prompt,
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=28,
    generator=torch.Generator(device=device).manual_seed(0),
).images[0]

These are quickstart prompts, one per task. The model was trained on many phrasings of each task, so you can adapt them:

• Semantic segmentation: list any classes with an RGB colour each. Distinct, saturated colours work best. Pixels outside the listed classes come out black.
• Referring segmentation: replace the description and the paint colour.
• Instance segmentation: prompt one class at a time; replace every car in the prompt with any noun phrase. The left-to-right color ordering helps maintain greater separation between assigned colors. The model may not follow it exactly, which is fine: each instance only needs its own distinct colour.
• Depth and surface normals: use the prompts as they are. Depth is relative inverse depth, with closer points brighter. Normals follow the encoding stated in the prompt.
• Image editing: keep the [TASK: IMAGE_EDITING] tag and write the edit as a plain instruction.
• Generation: a plain description without a tag and without an input image.

🧩 LoRA adapters

ablation_models/f2kbase4b_lora_zeroshot_ablations/ contains the LoRA adapters of the paper's LoRA table, trained on FLUX.2 [klein] Base 4B over the zero-shot data: ranks 64, 128 and 256 at 25K, 37.5K, 50K, 75K and 100K steps, and ranks 512 and 1024 at 37.5K steps. manifest.md lists every adapter. Each adapter loads into the unmodified 4B base pipeline:

import torch
from diffusers import Flux2KleinPipeline

pipe = Flux2KleinPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-base-4B", torch_dtype=torch.bfloat16
)
pipe.load_lora_weights(
    "FunAILab/Open-Vision-Banana",
    subfolder="ablation_models/f2kbase4b_lora_zeroshot_ablations/r64/checkpoint-75000",
    weight_name="pytorch_lora_weights.safetensors",
)

Prompts and sampling settings are the same as above.

📄 License

The released weights are derived from FLUX.2 [klein] Base and are distributed under the FLUX Non-Commercial License.

📚 Citation

If you use Open Vision Banana in your research, please cite:

@misc{jadon2026openvisionbananafully,
      title={Open Vision Banana: A Fully Open Recipe for Generative Visual Perception},
      author={Arpit Jadon and Valentinos Pariza and Jona Ruthardt and Ryousuke Yamada and A. Sophia Koepke and Andrei Bursuc and Yuki M. Asano},
      year={2026},
      eprint={2611.15844},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2611.15844},
}