README.md

---
license: apache-2.0
base_model: robbyant/lingbot-vision-vit-large
library_name: mlx
pipeline_tag: image-feature-extraction
tags:
  - mlx
  - vision-transformer
  - image-feature-extraction
  - self-supervised-learning
  - dense-prediction
---

# LingBot-Vision ViT-L/16 — MLX (Swift) weights

Unofficial **MLX** conversion of the
[LingBot-Vision](https://github.com/robbyant/lingbot-vision) ViT-L/16
self-supervised backbone, for running natively on Apple Silicon via
[mlx-swift](https://github.com/ml-explore/mlx-swift). Same weights as
[`robbyant/lingbot-vision-vit-large`](https://huggingface.co/robbyant/lingbot-vision-vit-large),
re-laid-out for the
[MLXLingBotVision](https://github.com/mnmly/mlx-swift-LingBot-Vision) Swift
package.

- **Files:** `model.safetensors` (fp32, ~1.13 GB) + `config.json` (architecture
  parameters read by the Swift loader).
- **Architecture:** ViT-L/16 — embed 1024 / depth 24 / heads 16, patch size 16,
  4 storage (register) tokens, axial 2D RoPE, LayerScale, fused-QKV attention
  with a masked K-bias.

## Not affiliated / not endorsed

This is an unofficial community conversion. It is **not** affiliated with or
endorsed by the LingBot-Vision authors (Ant Group). All credit for the model and
the training method belongs to the original authors.

## Changes vs. the original checkpoint

The conversion
([`scripts/convert.py`](https://github.com/mnmly/mlx-swift-LingBot-Vision/blob/main/scripts/convert.py))
is numerically neutral — it only re-lays-out the weights for MLX Swift:

- Unwrapped the state-dict wrappers and stripped the `_orig_mod.` / `backbone.`
  key prefixes.
- Baked the fused-QKV `bias_mask` into `attn.qkv.bias` (zeroing the K third of
  the bias) and dropped the mask buffer, so the Swift side is a plain fused
  `Linear`.
- Dropped the MIM-only `mask_token` (a no-op at eval time — the forward uses
  `cls_token + 0 * mask_token`).
- Saved fp32 `safetensors` with keys matching the Swift module tree, plus a
  `config.json`.
- Conv2d patch-embed weights are transposed PyTorch NCHW → MLX NHWC by the Swift
  loader at load time.

## Parity

Verified end-to-end against the Python reference (fp32): patch-token cosine
`0.9999987`, maxAbs `0.010`, meanAbs `0.0004`; CLS-token cosine `0.99999624`.

## Usage

With the [MLXLingBotVision](https://github.com/mnmly/mlx-swift-LingBot-Vision)
Swift package:

```swift
import MLXLingBotVision

let session = try LingBotVisionSession.load(
    SessionConfig(modelDirectory: URL(fileURLWithPath: "/path/to/lingbot-vision-vit-large-mlx"),
                  dtype: .float16))

let out = try session.features(imageURL: imageURL, size: 512)   // cls / storage / patch tokens
let cg  = try session.pcaCGImage(imageURL: imageURL, size: 512) // PCA RGB visualization
```

CLI:

```bash
lbv-tool --model /path/to/lingbot-vision-vit-large-mlx --image example.png --out pca.png --size 512
```

## Credits & citation

Original model by Zelin Fu, Bin Tan, Changjiang Sun, Shaohui Liu, Kecheng Zheng,
Yinghao Xu, Xing Zhu, Yujun Shen, Nan Xue. LingBot-Vision acknowledges DINOv2 and
DINOv3.

```bibtex
@article{lingbot-vision2026,
  title={Vision Pretraining for Dense Spatial Perception},
  author={Fu, Zelin and Tan, Bin and Sun, Changjiang and Liu, Shaohui and Zheng, Kecheng and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},
  year={2026}
}
```

## License

Apache 2.0 — same as the original checkpoint. See [`LICENSE`](LICENSE).