mnmly/lingbot-vision-vit-large-mlx

LingBot-Vision ViT-L/16 — MLX (Swift) weights

Unofficial MLX conversion of the LingBot-Vision ViT-L/16 self-supervised backbone, for running natively on Apple Silicon via mlx-swift. Same weights as robbyant/lingbot-vision-vit-large, re-laid-out for the MLXLingBotVision Swift package.

• Files: model.safetensors (fp32, ~1.13 GB) + config.json (architecture parameters read by the Swift loader).
• Architecture: ViT-L/16 — embed 1024 / depth 24 / heads 16, patch size 16, 4 storage (register) tokens, axial 2D RoPE, LayerScale, fused-QKV attention with a masked K-bias.

Not affiliated / not endorsed

This is an unofficial community conversion. It is not affiliated with or endorsed by the LingBot-Vision authors (Ant Group). All credit for the model and the training method belongs to the original authors.

Changes vs. the original checkpoint

The conversion (scripts/convert.py) is numerically neutral — it only re-lays-out the weights for MLX Swift:

• Unwrapped the state-dict wrappers and stripped the _orig_mod. / backbone. key prefixes.
• Baked the fused-QKV bias_mask into attn.qkv.bias (zeroing the K third of the bias) and dropped the mask buffer, so the Swift side is a plain fused Linear.
• Dropped the MIM-only mask_token (a no-op at eval time — the forward uses cls_token + 0 * mask_token).
• Saved fp32 safetensors with keys matching the Swift module tree, plus a config.json.
• Conv2d patch-embed weights are transposed PyTorch NCHW → MLX NHWC by the Swift loader at load time.

Parity

Verified end-to-end against the Python reference (fp32): patch-token cosine 0.9999987, maxAbs 0.010, meanAbs 0.0004; CLS-token cosine 0.99999624.

Usage

With the MLXLingBotVision Swift package:

import MLXLingBotVision

let session = try LingBotVisionSession.load(
    SessionConfig(modelDirectory: URL(fileURLWithPath: "/path/to/lingbot-vision-vit-large-mlx"),
                  dtype: .float16))

let out = try session.features(imageURL: imageURL, size: 512)   // cls / storage / patch tokens
let cg  = try session.pcaCGImage(imageURL: imageURL, size: 512) // PCA RGB visualization

CLI:

lbv-tool --model /path/to/lingbot-vision-vit-large-mlx --image example.png --out pca.png --size 512

Credits & citation

Original model by Zelin Fu, Bin Tan, Changjiang Sun, Shaohui Liu, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Nan Xue. LingBot-Vision acknowledges DINOv2 and DINOv3.

@article{lingbot-vision2026,
  title={Vision Pretraining for Dense Spatial Perception},
  author={Fu, Zelin and Tan, Bin and Sun, Changjiang and Liu, Shaohui and Zheng, Kecheng and Xu, Yinghao and Zhu, Xing and Shen, Yujun and Xue, Nan},
  year={2026}
}

License

Apache 2.0 — same as the original checkpoint. See LICENSE.