README.md

---
license: apache-2.0
base_model: robbyant/lingbot-map
tags:
- coreml
- 3d-reconstruction
- camera-pose-estimation
- depth-estimation
- apple-silicon
- metal
- fp32
- fp16
pipeline_tag: depth-estimation
library_name: coremltools
---

# Apple CoreML LingBot-MAP Model Packages

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![GitHub](https://img.shields.io/badge/GitHub-uqer1244%2FcoreML__lingbot--map-black)](https://github.com/uqer1244/coreML_lingbot-map)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-uqer1244%2FcoreML__lingbot--map-yellow)](https://huggingface.co/uqer1244/coreML_lingbot-map)

This repository hosts converted Apple CoreML `.mlpackage` model bundles for **LingBot-MAP / Geometric Context Transformer (GCT)**, optimized for real-time 3D spatial mapping, camera pose estimation, and metric depth prediction on macOS and iOS (Apple Silicon M1/M2/M3/M4 & Apple Neural Engine).

---

## 🔗 Links & Resources

- **GitHub Repository**: [uqer1244/coreML_lingbot-map](https://github.com/uqer1244/coreML_lingbot-map)
- **Hugging Face Repository**: [uqer1244/coreML_lingbot-map](https://huggingface.co/uqer1244/coreML_lingbot-map)
- **MLX Repository**: [uqer1244/mlx_lingbot-map](https://github.com/uqer1244/mlx_lingbot-map)
- **Original Model Weights**: [robbyant/lingbot-map](https://huggingface.co/robbyant/lingbot-map)

---

## 📦 Hosted CoreML Model Packages

| Package Name | Input Spec | Output Spec | Description |
| :--- | :--- | :--- | :--- |
| **`Aggregator.mlpackage`** | `images`: `[1, 2, 3, 518, 378]` | `t0..t3`: `[1, 2, 1005, 2048]` | Feature trunk backbone for spatio-temporal embeddings |
| **`StatefulCameraHead.mlpackage`** | `t0..t3`, `k_cache_in`, `v_cache_in`, `frame_idx_in` | `pred_pose_enc`, `k_cache_out`, `v_cache_out`, `frame_idx_out` | Stateful KV Cache camera pose head (Rank-3 state tensors) |
| **`DepthHead.mlpackage`** | `t0..t3`, `images` | `depth`: `[1, 2, 518, 378, 1]`, `confidence` | Metric depth map ($518 \times 378$) & confidence estimator |
| **`Encoder.mlpackage`** | `images`: `[1, 2, 3, 518, 378]` | `tokens`: `[1, 2, 1005, 1024]` | ViT patch embedder (Integrated inside Aggregator) |

---

## 🛠️ Technical Specifications

- **Preprocessing**: Aspect-preserving portrait center-crop ($518 \times 378$) preserving $1:1$ physical pixel geometry.
- **State Management**: Rank-3 KV Cache state tensors ($256 \times 64 \times 128$) with dynamic frame index tracking.
- **Pose Transformation**: Closed-form SE3 matrix inversion ($W2C \rightarrow C2W$) and cumulative camera transformation ($C2W_k = C2W_{k-1} \times \Delta C2W_k$).

---

## 💻 Download & Usage Guide

### 1. Download via Hugging Face Hub (Python)

```bash
pip install huggingface_hub
```

```python
from huggingface_hub import snapshot_download

# Download all CoreML model packages to local models/ directory
snapshot_download(repo_id="uqer1244/coreML_lingbot-map", local_dir="./models")
```

### 2. Standalone Python MVP Runner

Clone the GitHub repository and run inference:

```bash
git clone https://github.com/uqer1244/coreML_lingbot-map.git
cd coreML_lingbot-map/LingBot-CoreML-MVP

pip install coremltools numpy opencv-python matplotlib viser
python3 run_coreml_mvp.py
```

### 3. Swift Native Xcode Integration

Add `LingBot-CoreML-Runtime` as a Swift Package Manager dependency in Xcode:

```swift
import LingBotCoreMLRuntime

let runtime = StreamingRuntime(config: .default)
try runtime.loadModels(from: modelDirectoryURL)
```

---

## 📄 License

This repository is licensed under the Apache 2.0 License.