Mirror of mlboydaisuke/RF-DETR-CoreAI

.gitattributes

@@ -33,3 +33,21 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+demo_coco_252219.jpg filter=lfs diff=lfs merge=lfs -text
+demo_coco_cats.jpg filter=lfs diff=lfs merge=lfs -text
+rfdetr-large_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-medium_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-nano_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-small_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+split/rfdetr-nano_backbone.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+split/rfdetr-nano_head.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+split/rfdetr-medium_backbone.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+split/rfdetr-medium_head.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-seg-nano_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-seg-small_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-seg-medium_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-seg-large_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-seg-xlarge_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+rfdetr-seg-2xlarge_float32.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
+demo_seg_coco_cats.jpg filter=lfs diff=lfs merge=lfs -text
+demo_seg_coco_252219.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,145 @@
+---
+license: apache-2.0
+base_model: roboflow/rf-detr
+tags:
+- coreai
+- aimodel
+- object-detection
+- rf-detr
+- detr
+- apple
+- ios
+- macos
+pipeline_tag: object-detection
+---
+
+> **Mirror** of [`mlboydaisuke/RF-DETR-CoreAI`](https://huggingface.co/mlboydaisuke/RF-DETR-CoreAI) — the canonical repo ([CoreAI Model Zoo](https://github.com/john-rocky/coreai-model-zoo)). Updates land there first.
+
+
+# RF-DETR — Core AI (`.aimodel`)
+
+[RF-DETR](https://github.com/roboflow/rf-detr) (Roboflow's real-time detection
+transformer, COCO-pretrained) converted to Apple **Core AI** for iOS 27 / macOS 27 —
+the answer to [apple/coreai-models#14](https://github.com/apple/coreai-models/issues/14).
+**DETR family = no NMS**: post-processing is one sigmoid + top-k.
+
+<p align="center"><img src="demo_coco_cats.jpg" width="440" alt="RF-DETR medium on Core AI"></p>
+
+<!-- gen-cards:use-it begin id=rf-detr (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->
+## Use it
+
+▶️ **Run it (source)** — the [DetectCamera runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/DetectCamera)
+(real-time object detection on the zero-copy camera path):
+
+```bash
+git clone https://github.com/john-rocky/coreai-kit
+open coreai-kit/Examples/DetectCamera/DetectCamera.xcodeproj
+# → Run, then pick "Nano" in the model picker
+
+# agents / headless (macOS):
+cd coreai-kit/Examples/DetectCamera
+swift run detect-cli --model rf-detr --image Resources/gate_image.jpg
+```
+
+💻 **Build with it** — complete; the glue is kit API, copy-paste runs:
+
+```swift
+import CoreAIKitVision
+
+let detector = try await KitDetector(catalog: "rf-detr")
+let image = try ImageFile.load(imageURL)  // any image file → CGImage + EXIF orientation
+let detections = try await detector.detect(in: image.cgImage)
+// detections: [Detection] — label, score, normalized box (top-left origin)
+```
+
+The take-home is [`Examples/DetectCamera/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/DetectCamera/Sources/QuickStart.swift)
+— this exact code as one typed function, no UI; the CLI is an argument shell over it, and
+the GUI runs the same detector per camera frame on a zero-copy pixel-buffer fast path.
+Real time? Use `detect(in: CVPixelBuffer)` — vImage scales the frame with no CGImage
+round-trip; `CameraFeed` (kit API) streams the buffers.
+
+**Integration checklist**
+
+- SPM: `https://github.com/john-rocky/coreai-kit` → product **CoreAIKitVision**
+- Info.plist: `NSCameraUsageDescription` — only for the live camera; the snippet needs none
+- Entitlements: none needed
+- First run downloads the model — 0.1 GB (Mac) / 0.1 GB (iPhone) — then it loads from the
+  local cache (Application Support; progress via the `downloadProgress` callback)
+- Measure in Release — Debug is ~3× slower on per-token host work
+<!-- gen-cards:use-it end -->
+
+## Files
+
+| file | input | params | M4 Max GPU | iPhone 17 Pro GPU |
+|---|---|---|---|---|
+| `rfdetr-nano_float32.aimodel` | 384×384 | 30.5M | **8.6 ms** (~116 FPS) | **~25 ms (33–39 FPS live)** |
+| `rfdetr-small_float32.aimodel` | 512×512 | 32.1M | **12.0 ms** (~83 FPS) | — |
+| `rfdetr-medium_float32.aimodel` | 576×576 | 33.7M | **14.8 ms** (~68 FPS) | **56–63 ms (15–17 FPS live)** |
+| `rfdetr-large_float32.aimodel` | 704×704 | 33.9M | **19.1 ms** (~52 FPS) | — |
+
+iPhone numbers are end-to-end live-camera measurements from the
+[CoreAIKit DetectCamera example](https://github.com/john-rocky/coreai-kit)
+(Release; zero-copy capture pipeline — AVCaptureVideoPreviewLayer display,
+hardware-scaled 32BGRA buffers, vImage preprocessing overlapped with GPU
+inference). Peak measured 39.6 FPS ≈ the nano model ceiling; sustained
+max-load throughput drops on a hot chassis (thermal).
+
+fp32 is the ship dtype: it gates **detection-set exact** vs the PyTorch fp32 reference on
+CPU and GPU (per confident detection: same class, IoU ≥ 0.999 measured, score within 2e-3),
+and fp16 only bought ~7% latency on M4 Max while adding near-tie ranking noise.
+
+## Graph contract
+
+```
+input  "image"  [1, 3, R, R]  float32, RGB in [0, 1]  (ImageNet mean/std folded in-graph)
+output "dets"   [1, 300, 4]   boxes, cxcywh normalized to [0, 1]
+output "labels" [1, 300, 91]  raw class logits; column index = ORIGINAL COCO id (0 unused, 1=person … 17=cat … 90)
+```
+
+Python decode sketch (Swift is the same three steps):
+
+```python
+import numpy as np, coreai.runtime as rt
+
+model = await rt.AIModel.load(path, rt.SpecializationOptions.default())
+fn = model.load_function("main")
+out = await fn({"image": rt.NDArray(rgb01)})          # rgb01: [1,3,R,R] in [0,1]
+prob = 1 / (1 + np.exp(-out["labels"].numpy()[0]))    # [300, 91]
+scores, classes = prob.max(-1), prob.argmax(-1)       # column index IS the COCO id
+boxes = out["dets"].numpy()[0]                        # cxcywh, multiply by image W/H
+keep = scores > 0.5                                   # done — no NMS
+```
+
+## RF-DETR-Seg (instance segmentation)
+
+`rfdetr-seg-{nano,small,medium,large,xlarge,2xlarge}_float32.aimodel` — same
+contract plus `masks [1, Q, R/4, R/4]`: per-query FULL-FRAME logit planes at
+stride 4 (host: sigmoid > 0.5; no ROI plumbing, no NMS). All six gate on CPU
+and GPU with binary-mask IoU 1.000 on stable scenes. M4 Max GPU:
+seg-nano 312² **10.7 ms** → seg-2xlarge 768² **59.1 ms**.
+
+<p align="center"><img src="demo_seg_coco_cats.jpg" width="440" alt="RF-DETR-Seg nano on Core AI"></p>
+
+## Split deployment (`split/`)
+
+`split/rfdetr-{nano,medium}_{backbone,head}.aimodel` separate the pure-ViT
+backbone (image → features) from the deformable head (features → dets/labels;
+position encodings baked in). The chain is bit-exact vs the monolith. Purpose:
+per-stage compute-unit preferences — e.g. backbone on the Neural Engine.
+Measured honestly: on iOS 27 beta the runtime still executes the backbone on
+the GPU delegate even under `.neuralEngine` preference (identical detection
+fingerprint, no ANE-compile pause), so today the monolith on GPU is the
+fastest config; the split exists so ANE placement can be adopted the moment
+the runtime honors it. Regenerate with `export_rf_detr.py --variant <v> --split`.
+
+## Conversion
+
+Exported with
+[`conversion/export_rf_detr.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_rf_detr.py)
+from `rfdetr==1.7.1` weights. The port surfaced four Core AI converter/runtime bugs
+(float-arg `arange` abort, int64-comparison buffer clobber, GPU-delegate
+floor/trunc/ceil = identity, cast-pair cancellation) — each worked around numerically
+identically; details and minimal repros in
+[zoo/rf-detr.md](https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/rf-detr.md).
+
+License: Apache-2.0 (upstream RF-DETR code and COCO-pretrained weights are Apache-2.0).

config.json

@@ -0,0 +1,7 @@
+{
+  "model_type": "coreai-aimodel",
+  "format": "aimodel",
+  "framework": "Apple Core AI (iOS 27 / macOS 27)",
+  "repository": "https://github.com/john-rocky/coreai-model-zoo",
+  "note": "Converted .aimodel bundles for Apple's Core AI framework. See README.md for the bundle layout and run instructions."
+}

rfdetr-large_float32.aimodel/main.hash

@@ -0,0 +1 @@
+�Q>�����76&�n�0���
za���ogF

rfdetr-large_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f2513ed3e3fcf3aaee863736268c6ef930bfb5e3ba017a619b98856f67160446
+size 121906905

rfdetr-large_float32.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

rfdetr-medium_float32.aimodel/main.hash

@@ -0,0 +1 @@
+��
{oW\J�2d׋���xReD��=�߮�

rfdetr-medium_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03f997017b6f575c4ac53264d78bfe168eb078520865441ed5c43da404dfaef2
+size 120911038

rfdetr-medium_float32.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

rfdetr-nano_float32.aimodel/main.hash

@@ -0,0 +1 @@
+]�Gέ*�O_�3S��

rfdetr-nano_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d8747cead2af04f5f9a3353f2a0a3fc0d5830f3979aa12a781c870c287e3e9b
+size 107962969

rfdetr-nano_float32.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

rfdetr-seg-2xlarge_float32.aimodel/main.hash

@@ -0,0 +1 @@
+xpEm�4?��Ma����a'���2֝{�T

rfdetr-seg-2xlarge_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7870456d91343f8c1316a04d61f2b9dacecc6127f1f2aef41932d69d7bc4547f
+size 140901149

rfdetr-seg-2xlarge_float32.aimodel/metadata.json

@@ -0,0 +1,7 @@
+{
+  "author" : "Roboflow (RF-DETR); Core AI export: coreai-model-zoo",
+  "assetVersion" : "2.0",
+  "description" : "RF-DETR real-time detection transformer (DINOv2-windowed backbone + deformable decoder, two-stage, no NMS). Input: RGB [0,1]; outputs: 300 cxcywh boxes + 91 COCO-id logit columns. https:\/\/github.com\/roboflow\/rf-detr",
+  "license" : "Apache-2.0",
+  "creationDate" : "20260612T091728Z"
+}

rfdetr-seg-large_float32.aimodel/main.hash

@@ -0,0 +1 @@
+L�=ӌs�����d��/mEW)t*�/����

rfdetr-seg-large_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cfc3d04d38c739dc083f6d6641cc7f52f6d1a450c5729742ab82f9bf48eeffd
+size 130856589

rfdetr-seg-large_float32.aimodel/metadata.json

@@ -0,0 +1,7 @@
+{
+  "license" : "Apache-2.0",
+  "author" : "Roboflow (RF-DETR); Core AI export: coreai-model-zoo",
+  "description" : "RF-DETR real-time detection transformer (DINOv2-windowed backbone + deformable decoder, two-stage, no NMS). Input: RGB [0,1]; outputs: 300 cxcywh boxes + 91 COCO-id logit columns. https:\/\/github.com\/roboflow\/rf-detr",
+  "assetVersion" : "2.0",
+  "creationDate" : "20260612T091624Z"
+}

rfdetr-seg-medium_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2ac79e1d257649e0080423cb1ae980a035057a73b168d741c3095b6dda7c983
+size 130128371

rfdetr-seg-medium_float32.aimodel/metadata.json

@@ -0,0 +1,7 @@
+{
+  "creationDate" : "20260612T091554Z",
+  "license" : "Apache-2.0",
+  "assetVersion" : "2.0",
+  "author" : "Roboflow (RF-DETR); Core AI export: coreai-model-zoo",
+  "description" : "RF-DETR real-time detection transformer (DINOv2-windowed backbone + deformable decoder, two-stage, no NMS). Input: RGB [0,1]; outputs: 300 cxcywh boxes + 91 COCO-id logit columns. https:\/\/github.com\/roboflow\/rf-detr"
+}

rfdetr-seg-nano_float32.aimodel/main.hash

@@ -0,0 +1 @@
+E�r�'
�����x�c�����

rfdetr-seg-nano_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45dc72ea270e1707019a1fa002a9eda278bc63a0b9bb8bb20df254d72521a07e
+size 122759405

rfdetr-seg-nano_float32.aimodel/metadata.json

@@ -0,0 +1,7 @@
+{
+  "creationDate" : "20260612T091354Z",
+  "author" : "Roboflow (RF-DETR); Core AI export: coreai-model-zoo",
+  "description" : "RF-DETR real-time detection transformer (DINOv2-windowed backbone + deformable decoder, two-stage, no NMS). Input: RGB [0,1]; outputs: 300 cxcywh boxes + 91 COCO-id logit columns. https:\/\/github.com\/roboflow\/rf-detr",
+  "assetVersion" : "2.0",
+  "license" : "Apache-2.0"
+}

rfdetr-seg-small_float32.aimodel/main.hash

@@ -0,0 +1 @@
+�.*����C@���p(f�-/��I�/Vq^

rfdetr-seg-small_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a12e2aa7908994164340b28ba3702866f52d2f0497be49cc2f055671065e0313
+size 123306024

rfdetr-seg-small_float32.aimodel/metadata.json

@@ -0,0 +1,7 @@
+{
+  "author" : "Roboflow (RF-DETR); Core AI export: coreai-model-zoo",
+  "assetVersion" : "2.0",
+  "creationDate" : "20260612T091531Z",
+  "license" : "Apache-2.0",
+  "description" : "RF-DETR real-time detection transformer (DINOv2-windowed backbone + deformable decoder, two-stage, no NMS). Input: RGB [0,1]; outputs: 300 cxcywh boxes + 91 COCO-id logit columns. https:\/\/github.com\/roboflow\/rf-detr"
+}

rfdetr-seg-xlarge_float32.aimodel/main.hash

@@ -0,0 +1,2 @@
+�Q�S��
+�w��̢����@�י�<Ƚ褶�

rfdetr-seg-xlarge_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2518453b6ce0aa8778883cca2e117dcfa924092d799aa3c021fc8bde8a4b6ee
+size 138729619

rfdetr-seg-xlarge_float32.aimodel/metadata.json

@@ -0,0 +1,7 @@
+{
+  "description" : "RF-DETR real-time detection transformer (DINOv2-windowed backbone + deformable decoder, two-stage, no NMS). Input: RGB [0,1]; outputs: 300 cxcywh boxes + 91 COCO-id logit columns. https:\/\/github.com\/roboflow\/rf-detr",
+  "assetVersion" : "2.0",
+  "author" : "Roboflow (RF-DETR); Core AI export: coreai-model-zoo",
+  "license" : "Apache-2.0",
+  "creationDate" : "20260612T091657Z"
+}

rfdetr-small_float32.aimodel/main.hash

@@ -0,0 +1 @@
+�����Lb��=�
p<��gֲ��|�(P���

rfdetr-small_float32.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7d90cfecadd4c1e6216fba23df501703cd7d067d6b20495937ce028509fc1e5
+size 114573344

rfdetr-small_float32.aimodel/metadata.json

@@ -0,0 +1,3 @@
+{
+  "assetVersion" : "2.0"
+}

split/rfdetr-medium_backbone.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53c303d1004daedf9b1f901bb3fc2f08aaac6617e6fa6848ceec359674658d91
+size 94282817

split/rfdetr-medium_backbone.aimodel/metadata.json

@@ -0,0 +1,4 @@
+{
+  "assetVersion" : "2.0",
+  "license" : "Apache-2.0"
+}

split/rfdetr-medium_head.aimodel/main.hash

@@ -0,0 +1 @@
+�#_%{����z�	̬Q~�x��<��*J`

split/rfdetr-medium_head.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:141dc5231b5f1a257bb8bfd7f47a17b909ccac517ee378bf893c8ad52a4a0f60
+size 26631231

split/rfdetr-medium_head.aimodel/metadata.json

@@ -0,0 +1,4 @@
+{
+  "assetVersion" : "2.0",
+  "license" : "Apache-2.0"
+}

split/rfdetr-nano_backbone.aimodel/main.hash

@@ -0,0 +1 @@
+e;��dS�T��ǜ�!4�h�+�ݹo\���+�

split/rfdetr-nano_backbone.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:653bb2d3645311d554d2ecb4c79cc32134d668982bf8ddb96f5c089aa48b2bbd
+size 93176619

split/rfdetr-nano_backbone.aimodel/metadata.json

@@ -0,0 +1,4 @@
+{
+  "assetVersion" : "2.0",
+  "license" : "Apache-2.0"
+}

split/rfdetr-nano_head.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba3ef1283562ea9b774d11949e52fc65b3001549f66233214a55c5c41f8c0769
+size 14790518

split/rfdetr-nano_head.aimodel/metadata.json

@@ -0,0 +1,4 @@
+{
+  "assetVersion" : "2.0",
+  "license" : "Apache-2.0"
+}