README.md

---
license: apache-2.0
language:
- en
tags:
- computer-vision
- self-supervised-learning
- vision-transformer
- image-feature-extraction
- dense-prediction
- depth-estimation
- semantic-segmentation
- pytorch
datasets:
- custom
library_name: pytorch
pipeline_tag: image-feature-extraction
---

# LingBot-Vision

**LingBot-Vision** is a family of self-supervised Vision Transformer backbones for dense spatial perception. The models are pretrained with masked boundary modeling, a boundary-centric objective that encourages spatially structured patch features while retaining strong semantic representations.

This Hugging Face repository stores a backbone-only PyTorch checkpoint as `model.pt`. It is intended for inference, feature extraction, PCA visualization, and downstream dense prediction research.

## Model Details

### Model Description

LingBot-Vision learns dense patch representations that preserve boundaries, shapes, and semantic regions. The backbone is trained from random initialization with self-supervised teacher-student pretraining. During training, teacher-discovered boundary tokens are forced into the masked set, and boundary tokens receive both semantic self-distillation and categorical boundary-field supervision.

The released model family includes:

- **LingBot-Vision-Giant:** ViT-g/16 backbone for highest-quality dense features.
- **LingBot-Vision-Large:** ViT-L/16 backbone for strong dense features and practical inference.
- **LingBot-Vision-Base:** ViT-B/16 backbone for balanced inference cost.
- **LingBot-Vision-Small:** ViT-S/16 backbone for lightweight demos and downstream use.

Each checkpoint contains backbone weights only. Training-time heads, optimizer states, and boundary-target generation components are not included.

- **Developed by:** Zelin Fu, Bin Tan, Changjiang Sun, Shaohui Liu, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Nan Xue
- **Model type:** Vision Transformer backbone for dense visual representation learning
- **License:** Apache 2.0

### Model Sources

- **Repository:** https://github.com/robbyant/lingbot-vision
- **Project Page:** https://technology.robbyant.com/lingbot-vision
- **Technical Report:** coming soon

### Related Models

- **LingBot-Vision-Giant:** https://huggingface.co/robbyant/lingbot-vision-vit-giant
- **LingBot-Vision-Large:** https://huggingface.co/robbyant/lingbot-vision-vit-large
- **LingBot-Vision-Base:** https://huggingface.co/robbyant/lingbot-vision-vit-base
- **LingBot-Vision-Small:** https://huggingface.co/robbyant/lingbot-vision-vit-small

## Uses

### Direct Use

- **Dense Feature Visualization:** Extract frozen patch tokens and visualize their PCA components.
- **Image Feature Extraction:** Use normalized patch tokens as spatial visual features.
- **Backbone Initialization:** Initialize downstream dense prediction models with LingBot-Vision weights.

### Downstream Use

- **Depth Estimation:** Frozen patch tokens expose spatial structure to lightweight dense readouts.
- **Semantic Segmentation:** Boundary-faithful features help align region transitions with object contours.
- **Video Object Segmentation:** Frozen features support training-free label propagation and token matching.
- **Depth Completion:** LingBot-Vision can serve as the visual encoder initialization for LingBot-Depth 2.0.

## How to Load

Install the LingBot-Vision inference repository and dependencies:

```bash
git clone https://github.com/robbyant/lingbot-vision.git
cd lingbot-vision

conda create -n lingbot-vision python=3.10 -y
conda activate lingbot-vision

python -m pip install -r requirements.txt
python -m pip install -e .
```

Load a pretrained backbone:

```python
import torch

from lbot_vision_infer import load_pretrained_backbone

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

backbone, embed_dim = load_pretrained_backbone(
    variant="large",
    device=device,
    dtype=dtype,
)

print(backbone.patch_size, embed_dim)
```

The `variant` argument can be `giant`, `large`, `base`, or `small`. You can also pass an explicit Hugging Face model repo or a local directory to `load_pretrained_backbone`.

## Technical Specifications

### Model Architecture

- **Backbone:** Vision Transformer with patch size 16
- **Released variants:** ViT-g/16, ViT-L/16, ViT-B/16, ViT-S/16
- **Output:** Normalized patch tokens from the frozen backbone
- **Checkpoint format:** Backbone-only `.pt` file stored as `model.pt`
- **Training objective:** Masked boundary modeling with self-distillation

### Software Requirements

- Python >= 3.10
- PyTorch >= 2.0.0
- huggingface_hub
- omegaconf

## Citation

```bibtex
@article{lingbot-vision2026,
  title={Vision Pretraining for Dense Spatial Perception},
  author={Fu, Zelin and Tan, Bin and Sun, Changjiang and Liu, Shaohui and Zheng, Kecheng and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},
  year={2026}
}
```

## Model Card Contact

- **Issues:** https://github.com/robbyant/lingbot-vision/issues
- **Email:** fuzelin.fzl@antgroup.com, xuenan.xue@antgroup.com