File size: 5,218 Bytes
33cf7d8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | ---
license: mit
base_model: egeorcun/lucida
base_model_relation: quantized
library_name: mlx
tags:
- mlx
- mlx-swift
- image-segmentation
- background-removal
- image-matting
- birefnet
pipeline_tag: image-segmentation
---
# Lucida-fp16 (MLX)
[egeorcun/lucida](https://huggingface.co/egeorcun/lucida) converted for **MLX / MLX-Swift** on Apple
Silicon. Lucida is a fine-tune of [ZhengPeng7/BiRefNet_HR](https://huggingface.co/ZhengPeng7/BiRefNet_HR)
for foreground matting / background removal, tuned toward the cases most open matting models handle
worst: camouflaged subjects, transparency, text and logos, print design, and illustration.
Single tier at **1024×1024**, the resolution Lucida was trained at.
## What was converted
`model.safetensors` — 687 tensors, **fp16** (~440 MB). Produced from the upstream fp32 checkpoint with
the `birefnet-convert` tool in
[xocialize/mlx-birefnet-swift](https://github.com/xocialize/mlx-birefnet-swift):
- checkpoint prefixes stripped; PyTorch-only buffers dropped (`relative_position_index`,
`num_batches_tracked` — 67 tensors)
- `decoder.conv_out1.0.{weight,bias}` → `decoder.conv_out1.{weight,bias}` (upstream wraps the final
conv in an `nn.Sequential`; the Swift port uses a bare conv)
- every 4-D conv weight transposed from PyTorch NCHW `(O, I, kH, kW)` to MLX NHWC `(O, kH, kW, I)`
- cast to fp16
The **architecture is unchanged** from BiRefNet_HR — the converted key set is byte-identical to both
upstream BiRefNet bases (754 tensors in → 687 out, zero keys or shapes differing), so this loads
through the stock BiRefNet graph (`swin_v1_l`, `mul_scl_ipt=cat`, `cxt_num=3`, `ASPPDeformable`).
## Verification
Gated against a PyTorch oracle at 1024², with the oracle's exact normalized input tensor injected on
both sides (so the comparison measures the model, not preprocessing):
| | |
|---|---|
| encoder stages (CPU stream) | maxAbs ≤ 8.3e-5 relative to range, cosine 1.0000000 |
| e2e logits | cosine **0.99998** |
| e2e sigmoid matte | cosine **0.99977** |
| foreground fraction | **0.0292**, matching the oracle's 0.0292 exactly |
**fp16 is the right dtype here, and bf16 is not.** At the same gate bf16 fails (sigmoid cosine
0.9958, foreground fraction drifting to 0.0272) — its 8-bit mantissa is too coarse for this graph.
No bf16 variant is published for that reason.
The fp32 conversion is bit-exact against the source, so any residual difference above is Apple-GPU
fp32 accumulation, not conversion loss.
## Usage (MLX-Swift)
```swift
import MLXServeCore
import MLXBiRefNet
let engine = MLXServeEngine()
let id = try await engine.register(Lucida.registration, configuration: .lucida())
let resp = try await engine.run(MattingRequest(image: photo), package: id) as! MattingResponse
// resp.matte — soft-alpha grayscale PNG at the input's dimensions
```
Measured on an M-series Max: ~0.38 s per 1024² matte, ~0.6 GB resident + ~14 GB peak transient.
## Choosing between Lucida and BiRefNet
Lucida is **not** a drop-in replacement — the two lead on different material. On the author's own
203-image benchmark (MAE, lower is better):
| Lucida is better at | BiRefNet-HR is better at |
|---|---|
| camouflage 0.0270 vs 0.0752 · print/design 0.0235 vs 0.0544 · text & logos 0.0091 vs 0.0207 · illustration 0.0092 vs 0.0157 · transparency 0.0358 vs 0.0687 | hair 0.0048 vs 0.0093 · thin structures 0.0196 vs 0.0322 · complex edges 0.0385 vs 0.0484 |
Overall on that set: 0.0257 vs 0.0346. So route photographic portrait/hair work to BiRefNet and
non-photographic, hard-edge, text or illustration work here.
> Those figures are the **author's own** — 203 self-selected images, 9 self-defined categories, MAE
> only (which flatters soft-alpha smoothness), and the published table names "Lucida-v7" while the
> source repo ships one unversioned checkpoint. Treat as directional, not decisive.
Also note the author's **colour-decontamination and edge-refinement passes are not part of the
checkpoint** — they live in [their GitHub repo](https://github.com/egeorcun/lucida). This conversion
provides the network only, emitting raw sigmoid alpha.
## License & provenance
MIT, following upstream's declaration for both the weights and the code.
One thing to be aware of before commercial use: the upstream card states the training data
"mix[es] MIT-licensed and research-only datasets" — P3M-10k, COD10K and DIS5K, alongside the
CC-BY-4.0 [ToonOut](https://huggingface.co/datasets/joelseytre/toonout) set. That is a question about
the data the model was derived from rather than about the MIT grant we received, but it is worth your
own review rather than inheriting the label uncritically.
## Credits
- **Lucida** — [egeorcun](https://huggingface.co/egeorcun) ([code + benchmark](https://github.com/egeorcun/lucida))
- **BiRefNet** — [ZhengPeng7](https://huggingface.co/ZhengPeng7) ([paper](https://arxiv.org/abs/2401.03407), [code](https://github.com/ZhengPeng7/BiRefNet))
- **MLX-Swift port** — [xocialize/mlx-birefnet-swift](https://github.com/xocialize/mlx-birefnet-swift), whose vendored core derives from [mnmly/mlx-swift-BiRefNet](https://github.com/mnmly/mlx-swift-BiRefNet) (MIT)
|