- Example Output
- ReasonSeg Benchmark Results
- Installation & Quick Start
- Limitations
- How the LIS1 Architecture Works
- Step 1: Causal Riddle-Before-Image Prompt Construction
- Step 2: Single-Pass Dual-Scale Vision Extraction (With LRU Caching)
- Step 3: LLM Forward Pass & Multi-Layer Feature Readout
- Step 4: Cross-Attention & Dynamic Feature Modulation
- Step 5: Coarse-to-Fine Spatial Mask Decoding
- Step 6: Post-Processing & Two-Stage Training
- Step 1: Causal Riddle-Before-Image Prompt Construction
- Training and Benchmark Commands
LIS1: Single-Pass Visual Reasoning & Referring Image Segmentation
LIS1 (Linguistic Image Segmenter) is a fast, non-autoregressive visual reasoning and referring image segmentation model family built on Qwen3.5 (0.8B, 2B, 4B, 9B, and 27B).
Unlike traditional reasoning segmentation models (such as LISA) that generate text tokens autoregressively and rely on a separate heavyweight SAM backbone, LIS1 performs both complex commonsense/riddle reasoning and crisp pixel-level mask decoding in a single forward pass using Qwen3.5's native Vision Transformer and LLM hidden statesβwith zero autoregressive token generation and no external SAM encoder.
Example Output
Below is a single-image inference example (image1.jpg to result_0_boxes_and_masks.png) generated by LIS1-0.8B in 4-bit quantization (example.py) for three simultaneous queries combining direct object names and functional riddles:
"toast"(Red β score0.75)"red sauce"(Green β score0.80)"something you can drink"(Blue β score0.88)
Input Image (image1.jpg) |
Rendered Output (result_0_boxes_and_masks.png) |
|---|---|
![]() |
![]() |
ReasonSeg Benchmark Results
All LIS1 models were trained exclusively on the RefCOCO dataset by lichengunc, which includes RefCOCO, RefCOCO+, and RefCOCOg (with 35% synthetic functional riddle augmentation). They were evaluated zero-shot on the ReasonSeg benchmark, without seeing ReasonSeg training images, using the ReasonSeg validation split and ReasonSeg test split.
Each repository provides two checkpoint variants (in both .safetensors and .pt formats):
noLora(Stage 1): Trained mask head with the base Qwen3.5 model kept fully frozen.forLora(Stage 2): Trained mask head + lightweight PEFT LoRA adapter (r=8,alpha=16) applied only to the top 4 LLM layers.
| Method | Val overall gIoU | Val overall cIoU | Test short query gIoU | Test short query cIoU | Test long query gIoU | Test long query cIoU | Test overall gIoU | Test overall cIoU |
|---|---|---|---|---|---|---|---|---|
| OVSeg (Liang et al., 2023) | 28.5 | 18.6 | 18.0 | 15.5 | 28.7 | 22.5 | 26.1 | 20.8 |
| GRES (Liu et al., 2023b) | 22.4 | 19.9 | 17.6 | 15.0 | 22.6 | 23.8 | 21.3 | 22.0 |
| X-Decoder (Zou et al., 2023) | 22.6 | 17.9 | 20.4 | 11.6 | 22.2 | 17.5 | 21.7 | 16.3 |
| SEEM (Zou et al., 2023b) | 25.5 | 21.2 | 20.1 | 11.5 | 25.6 | 20.8 | 24.3 | 18.7 |
| LISA-7B | 44.4 | 46.0 | 37.6 | 34.4 | 36.6 | 34.7 | 36.8 | 34.1 |
| LISA-7B (ft) | 52.9 | 54.0 | 40.6 | 40.6 | 49.4 | 51.0 | 47.3 | 48.4 |
| LISA-13B | 48.9 | 46.9 | 39.9 | 43.3 | 46.4 | 46.5 | 44.8 | 45.8 |
| LISA-13B (ft) | 56.2 | 62.9 | 44.3 | 42.0 | 54.0 | 54.3 | 51.7 | 51.1 |
| LISA-Llama2-13B (ft) | 60.0 | 67.8 | 43.9 | 45.8 | 54.0 | 53.8 | 51.5 | 51.3 |
| LIS1-0.8B noLora | 31.7 | 32.5 | 28.6 | 23.7 | 27.3 | 30.6 | 27.5 | 29.6 |
| LIS1-0.8B forLora | 32.4 | 31.7 | 30.4 | 23.9 | 29.1 | 29.6 | 29.4 | 28.6 |
| LIS1-2B noLora | 37.4 | 33.1 | 35.1 | 27.2 | 34.5 | 33.6 | 34.6 | 32.6 |
| LIS1-2B forLora | 37.5 | 33.1 | 35.5 | 27.1 | 34.6 | 34.3 | 34.8 | 33.1 |
| LIS1-4B noLora | 42.3 | 38.3 | 39.9 | 29.9 | 43.5 | 40.5 | 42.8 | 38.7 |
| LIS1-4B forLora | 43.5 | 39.6 | 40.1 | 29.7 | 42.9 | 39.2 | 42.4 | 37.7 |
| LIS1-9B noLora | 45.2 | 38.3 | 43.1 | 35.7 | 42.7 | 37.3 | 42.8 | 37.0 |
| LIS1-9B forLora | 45.1 | 40.0 | 40.8 | 33.3 | 44.7 | 40.9 | 43.9 | 39.6 |
| LIS1-27B noLora | 54.1 | 54.3 | 46.9 | 44.0 | 49.9 | 49.8 | 49.3 | 49.0 |
| LIS1-27B forLora | 54.0 | 54.8 | 47.1 | 44.6 | 50.7 | 51.4 | 50.0 | 50.4 |
Other results come from evaluation done by the team behind LISA
LIS1 2B and bigger had better results than Ultralytics MobileSAM and YOLOE Open Vocabulary in iternal evaluation.
Installation & Quick Start
1. Install Dependencies
pip install transformers peft safetensors pillow opencv-python numpy pyarrow torch torchvision
(Optional) For 4-bit or 8-bit quantization and faster Qwen3.5 linear-attention kernels on CUDA GPUs:
pip install bitsandbytes accelerate causal-conv1d flash-linear-attention
2. Minimal Inference Code
See example.py for the complete minimal script that produces the rendered image above. Make sure LIS1ModelCode.py and lis1_infer.py are in your working directory:
from PIL import Image
from lis1_infer import LIS1Inferencer
# 1. Initialize the inferencer
inferencer = LIS1Inferencer(debug=True)
# 2. Load the base Qwen3.5 model, LIS1 mask head (.safetensors or .pt), and optional LoRA adapter
# Swap "0.8B" with "2B", "4B", "9B", or "27B" depending on the model size you use.
inferencer.load_model(
qwen_path="./Qwen3.5-4B",
lis_model_path="./LIS1-4B-forLora.safetensors",
lora_adapter="./LIS1-4B-LoraAdapter", # Pass None if using a *-noLora checkpoint
device="cuda",
max_pixels=524288,
load_in_4bit=True, # Set False for full bfloat16, or load_in_8bit=True for 8-bit
)
# 3. Prepare images and natural language / riddle queries
images = [Image.open("image1.jpg").convert("RGB")]
queries = [["toast", "red sauce", "something you can drink"]]
# 4. Run single-pass segmentation
outputs = inferencer.run_model(
images=images,
queries=queries,
threshold=0.7,
isolate_queries=True,
cache_vision=True,
exclusive_masks=True,
)
# 5. Render masks, contours, bounding boxes, and non-overlapping labels
rendered = inferencer.render_image(images=images, outputs=outputs)
rendered[0].save("result_0_boxes_and_masks.png")
# 6. Free GPU/CPU memory when finished
inferencer.offload_model()
Limitations
- The model was trained and tested in English, but it should work, better or worse, in other languages Qwen3.5 knows too.
- The model has same bias, world understanding and knowledge cutoff as Qwen3.5.
- Masks may not be clean or have "islands" in other parts of the image.
- The model is designed to draw mask over one object, it may not work to for example segment all cars in an image separately.
- The model works best with objective appearance descriptions. It might not work with trademark names of things. It should not be used with abstract concepts like "friend", "evil" etc. and in case these are used it will usually draw mask over whatever the concept could be applied to, not if the object or person represents the abstract trait.
How the LIS1 Architecture Works
LIS1 turns a pretrained Qwen3.5 vision-language model into a single-pass, multi-object pixel segmenter through six sequential stages:
Step 1: Causal Riddle-Before-Image Prompt Construction
In causal language models, a token can only attend to tokens that appear earlier in the sequence. LIS1 formats the multimodal chat messages in a specific causal order:
- System Prompt: Instructs the model to act as a single-pass visual reasoning and segmentation model.
- User Turn (JSON Queries First, Image Second): The target descriptions or riddles are serialized into a JSON block before the
<|image_pad|>visual tokens. Because the text precedes the image, every visual patch token inside the LLM can attend directly to the user's riddles across all 24β64 layers. - Prefilled Assistant Turn: Instead of generating tokens one by one, LIS1 prefills the assistant response with a JSON structure wrapping each query inside Qwen's native grounding delimiters:
<|object_ref_start|>{description}<|object_ref_end|>. Crossing into the assistant turn and reading<|object_ref_end|>activates Qwen's internal spatial localization circuit in a single forward pass.
Step 2: Single-Pass Dual-Scale Vision Extraction (With LRU Caching)
When an image enters Qwen3.5's Vision Transformer (backbone.visual), a forward hook captures two spatial feature representations from the same pass without running an external segmentation backbone:
- Fine Unmerged ViT Features (
f_fine): Captured at 1/16 resolution before spatial patch merging, preserving sharp object edges and thin structures. - Coarse Merged ViT Features (
f_coarse): Captured at 1/32 resolution after 2x2 spatial merging (spatial_merge_size = 2). - Vision Caching: When segmenting multiple isolated queries on the same image (
isolate_queries=True, cache_vision=True), LIS1 runs the ViT encoder only once, caches the reconstructed 2D spatial grids (f_coarse,f_fine) and merged visual embeddings in an LRU cache, and injects the cached embeddings directly into the LLM token stream viamasked_scatter.
Step 3: LLM Forward Pass & Multi-Layer Feature Readout
The sequence of text and visual tokens passes once through the Qwen3.5 hybrid backbone (which interleaves Gated DeltaNet linear-attention layers with full self-attention layers; the lm_head is dropped completely). Forward hooks on the last 4 decoder layers extract two sets of representations:
- Riddle-Conditioned Visual Grid (
f_llm_coarse): The hidden states at all<|image_pad|>positions are averaged across the top 4 layers and reshaped back into a 2D spatial grid that now carries full multimodal reasoning context. - Multi-Layer Object Decision Vectors (
queries): For each of the K target objects, the hidden states at the<|object_ref_end|>token position are extracted from the last 4 layers ([K, 4, llm_dim]).
(Note on Quantization): When 4-bit or 8-bit bitsandbytes quantization is enabled, LIS1 automatically excludes the ViT encoder (visual) and Qwen3.5's tiny Gated DeltaNet decay gates (in_proj_a, in_proj_b) from quantization so spatial accuracy and linear-attention stability are preserved.
Step 4: Cross-Attention & Dynamic Feature Modulation
Inside MultiScaleMaskHead (kept in float32 for crisp boundary decoding):
- Layer Fusion: The 4 captured LLM layers for each query vector are normalized via per-layer
LayerNormand combined using learned softmax weights into a single decision vector per object ([K, llm_dim]). - Visual Context Fusion: The coarse ViT grid (
f_coarse) and the LLM-contextualized visual grid (f_llm_coarse) are projected tohidden_dim = 256using 1x1Conv2d + GroupNorm + GELUblocks and combined via a learnedtanh-gated residual connection. - 2D Cross-Attention: The K object queries are projected to
256dimensions and refined via 4-head cross-attention over the coarse 2D visual tokens, which are augmented with parameter-free 2D sine-cosine positional embeddings. - Calibrated IoU Score Prediction: A 2-layer MLP (
score_head) reads the refined query vector to predict an intrinsic mask confidence / IoU score in [0, 1].
Step 5: Coarse-to-Fine Spatial Mask Decoding
All K object masks are decoded in parallel through a three-stage hierarchy:
- Coarse Stage (1/32 Scale): Each query computes a spatial dot-product cosine similarity map against both the ViT coarse features and the LLM visual features. Simultaneously, the query generates a channel-wise sigmoid gate to modulate the 2D coarse feature map. The modulated features and similarity map (
256 + 1channels) are fused with a 3x3 convolution and a residual refinement block. - Fine Skip-Connection Stage (1/16 Scale): The coarse features are bilinearly upsampled 2x and concatenated with the fine unmerged ViT skip features (
f_fine, projected to128channels and gated by the query vector). Two consecutive 3x3 residual refinement blocks sharpen the object boundaries. - Full-Resolution Output Stage: A learned 2x transposed convolution (
ConvTranspose2dto 1/8 scale) followed by a 1x1 convolution produces mask logits, which are bilinearly interpolated to the exact original image dimensions(H, W).
Step 6: Post-Processing & Two-Stage Training
- Inference Post-Processing: Pixel probabilities are thresholded, cleaned with connected-component filtering (
filter_mask_componentsto drop tiny stray noise islands while preserving small single-component objects), optionally disambiguated across queries via pixel-wise winner-takes-all (exclusive_masks=True), and converted into both binary masks and COCO polygons/bounding boxes. The final confidence score blends the IoU score-head output (50%) with the mean foreground pixel probability (50%). - Two-Stage Training:
- Stage 1 (
noLora): Freezes all of the Qwen3.5 backbone and trains onlyMultiScaleMaskHeadfor 5 epochs using a combination of Binary Cross-Entropy (1.0), Sigmoid Focal Loss (2.0), Soft Dice Loss (2.0), and MSE IoU Score Calibration Loss (0.5). - Stage 2 (
forLora): Keeps the first 85-95% of the LLM layers completely frozen (preserving pretrained world knowledge and riddle-solving capabilities) and attaches low-rank LoRA adapters (r=8) only to the top 4 LLM layers (q_proj, k_proj, v_proj, o_proj, in_proj_qkv, out_proj) for 1β3 epochs of joint fine-tuning.
- Stage 1 (
Training and Benchmark Commands
The commands below assume the project scripts are in the current directory and the datasets are available locally. Replace {SIZE} with a Qwen3.5 model size such as 0.8B, 2B, 4B, 9B, or 27B; update the model and dataset paths for your setup.
Stage 1: Train the segmentation head
python3 train.py \
--stage 1 \
--model-path ./Qwen3.5-{SIZE} \
--data-root ./ref_coco_lichengunc \
--dataset all \
--riddle-prob 0.35 \
--epochs 5 \
--batch-size 4 \
--max-objects 4 \
--lr-head 2e-4 \
--output-dir ./checkpoints_{SIZE}
Stage 2: Fine-tune with LoRA
Resume from the Stage 1 checkpoint you want to continue from (the example uses epoch 5):
python3 train.py \
--stage 2 \
--model-path ./Qwen3.5-{SIZE} \
--data-root ./ref_coco_lichengunc \
--dataset all \
--riddle-prob 0.35 \
--resume-ckpt ./checkpoints_{SIZE}/segmenter_stage1_epoch5.pt \
--lora-top-layers 4 \
--lora-r 8 \
--lora-alpha 16 \
--epochs 3 \
--batch-size 4 \
--lr-head 1e-4 \
--lr-lora 2e-5 \
--output-dir ./checkpoints_{SIZE}
Evaluate on ReasonSeg
python3 benchmark_reasonseg.py \
--model-path ./Qwen3.5-{SIZE} \
--data-dir ./ReasonSeg_val \
--test-dir ./ReasonSeg_test \
--resume-ckpt ./checkpoints_{SIZE}/segmenter_stage2_epoch3.pt \
--method-name "LIS1-{SIZE} (Stage 2)" \
--out-md ./reasonseg_benchmark-{SIZE}.md
Use the Stage 1 checkpoint path instead to benchmark a noLora model. Stage 2 checkpoints require the corresponding LoRA adapter directory saved alongside the checkpoint; the path recorded in the checkpoint must resolve when running the benchmark. train.py defaults to CUDA when available, while the benchmark command can be run on CPU by adding --device cpu.
- Downloads last month
- 22

