File size: 5,660 Bytes
daa7522 9dc6402 daa7522 9dc6402 af461b0 9dc6402 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 | ---
license: cc-by-nc-nd-4.0
base_model:
- UmutKocasari/FaceAnything
pipeline_tag: depth-estimation
tags:
- coreml
- Apple
- MPS
---
# FaceAnything β CoreML (Apple Silicon)
[Link to GH Fork](https://github.com/AristidesAI/FaceAnything-CoreML)
Convert and run **Face Anything** natively on Apple Silicon (M-series Macs) using
CoreML with Metal GPU acceleration. No CUDA required.
The full 1.39B-parameter ViT-Giant model is converted to a single CoreML
`mlpackage` with fp16 precision (~2.5 GB), deployable on macOS 15+ and iOS 18+.
## What's Included
| File | Purpose |
|---|---|
| `coreml_scripts/convert_coreml.py` | Convert the PyTorch checkpoint β CoreML mlpackage |
| `coreml_scripts/run_coreml.py` | Full inference pipeline (masks β model β videos + PLY) |
| `coreml_models/FaceAnything_fp16.mlpackage` | Converted model (fp16, ~2.5 GB) |
| `coreml_models/FaceAnything_palettize_4bit.mlpackage` | Quantized model (4-bit, ~624 MB) |
## Requirements
- macOS 15+ with Apple Silicon (M1 or later)
- Python 3.11
- `venv_mps` or equivalent with `coremltools`, `torch`, `opencv-python`, `numpy`
Install from the existing `requirements_mps.txt`:
```bash
pip install -r requirements_mps.txt
```
## Quick Start
### 1. Convert the model (one-time)
```bash
# fp16 (2.5 GB, best quality)
python coreml_scripts/convert_coreml.py --resolution 504 --quantize none
# 4-bit palettized (~624 MB, very good quality)
python coreml_scripts/convert_coreml.py --resolution 504 --quantize palettize_4bit
# For vertical video (adjust width to match aspect ratio)
python coreml_scripts/convert_coreml.py --resolution 504 --width 280 --quantize none
```
The model's longest side is set to the `--resolution` value (must be divisible by
14, the ViT patch size). Use `--width` to set a non-square aspect ratio. The
output goes to `coreml_models/` by default.
### 2. Run inference
```bash
python coreml_scripts/run_coreml.py \
--input video.mov \
--output output_dir/ \
--render-size 1080 \
--fps 60
```
Options:
| Flag | Default | Description |
|---|---|---|
| `--input` | (required) | Video, image, or image folder |
| `--output` | (required) | Output directory |
| `--coreml-model` | `coreml_models/FaceAnything_fp16.mlpackage` | Path to mlpackage |
| `--render-size` | 1080 | Output canvas height |
| `--fps` | 60 | Output video frame rate |
| `--outputs` | `all` | Comma-separated: `videos`, `ply`, `grandtour` or `all` |
## Output Structure
```
output_dir/
βββ videos/
β βββ pointcloud.mp4 # RGB-colored 3D point cloud, sway orbit
β βββ depth.mp4 # Depth-colored (jet) 3D reconstruction
β βββ normals.mp4 # Surface normal-colored 3D
β βββ canonical.mp4 # Canonical face coordinates colored
β βββ tracks.mp4 # Temporally-consistent point tracks
β βββ grand_tour.mp4 # Morph through all modalities
βββ ply/
β βββ geometry/ # RGB-colored .ply per frame
β βββ depth/ # Depth-colored (jet) .ply per frame
β βββ normals/ # Normal-colored .ply per frame
β βββ canonical/ # Canonical-colored .ply per frame
β βββ tracks/ # Track-colored .ply per frame
βββ masks/ # Generated RVM foreground masks
βββ _frames_in/ # Extracted input frames
```
## Model Architecture
| Property | Value |
|---|---|
| Backbone | ViT-Giant (40 blocks, 1536-dim) |
| Parameters | 1.39B |
| Attn heads | 24 |
| Patch grid | 36 Γ 20 (for 504Γ280 input) |
| fp32 size | 5.6 GB |
| fp16 size | 2.5 GB |
| 4-bit size | 624 MB |
| Input | `(1, 3, 504, 280)` fp16, ImageNet-normalized |
| Outputs | depth, depth_conf, deformation, deformation_conf, intrinsics |
| Runtime | Metal GPU + AMX CPU; partial ANE for DPT heads |
## Conversion Notes
The model required significant patching to produce a CoreML-compatible graph:
| Original Op | Replacement |
|---|---|
| `torch.einsum` | Broadcasting |
| `F.scaled_dot_product_attention` | Manual QKα΅ Γ· βd Γ V |
| `torch.utils.checkpoint` (7 sites) | No-op passthrough |
| `einops.rearrange` (8 patterns) | Explicit `reshape` |
| `torch.chunk` (SwiGLU) | Explicit slicing |
| `int(positions.max())` (RoPE) | Fixed 8192-length cache |
| `addict.Dict` return type | Plain tuple |
| `upsample_bicubic2d.vec` | Registered as bilinear alias |
| `aten.alias` | Identity passthrough |
Multi-view features (reference view selection, camera tokens, global attention
alternation) are disabled for the single-view export (`alt_start = -1`).
## Quantization Options
| Method | Size | Deploy Target | Quality |
|---|---|---|---|
| `none` (fp16) | 2.5 GB | macOS 15 / iOS 18 | Reference |
| `palettize_4bit` | 624 MB | macOS 15 / iOS 18 | Very good |
| `palettize_8bit` | 1.2 GB | macOS 15 / iOS 18 | Excellent |
| `linear_8bit` | 1.2 GB | macOS 15 / iOS 18 | Excellent |
Palettization uses k-means weight clustering. The 4-bit model achieves ~75%
compression with negligible quality loss (validated numerically against fp32
PyTorch: depth max error < 0.01).
## Citation
```bibtex
@article{kocasari2026face,
title={Face Anything: 4D Face Reconstruction from Any Image Sequence},
author={Kocasari, Umut and Giebenhain, Simon and Shaw, Richard and Nie{\ss}ner, Matthias},
journal={arXiv preprint arXiv:2604.19702},
year={2026}
}
```
## License
This work is licensed under a
[Creative Commons Attribution-NonCommercial 4.0 International License](https://creativecommons.org/licenses/by-nc/4.0/).
CoreML adaptation is provided as-is under the same license. |