README.md

---
license: agpl-3.0
library_name: onnx
pipeline_tag: object-detection
tags:
  - yolov8
  - yolo-world
  - onnx
  - object-detection
  - open-images
  - open-vocabulary
  - in-browser
  - onnxruntime-web
---

# danmu-detector

Two ONNX detectors for in-browser furniture detection via `onnxruntime-web`,
meant to be run **together**. Built for Danmu, a local-first interior decoration
app — everything runs on-device, and works offline once the files are cached.

They are paired because they fail on *different* classes. Measured on a real
four-photo room containing 19 catalogued objects:

| | recall |
|---|---|
| `yolov8n-oiv7` alone | 7/19 (37%) |
| `yolov8s-worldv2-danmu` alone | 10/19 (53%) |
| **both** | **13/19 (68%)** |

The fixed-label model reliably finds monitors and windows. The open-vocabulary
one finds refrigerators, ceiling fans, wardrobes, lamps and curtains — none of
which the first detects **at all**: those class heads peak at 0.002–0.03 on real
room photos against 0.38–0.44 for classes that do fire, and they stay dead at
every model size (s, m and x all score the same 7/19 as nano). That is a
vocabulary limit, not a capacity one, which is what the second model fixes.

Nothing here depends on an external repository; the reference implementation
below is complete.

## Files

| File | Size | Output | Notes |
|---|---|---|---|
| `yolov8n-oiv7.onnx` | 14.2 MB | `1x605x8400` | YOLOv8 nano, Open Images V7, 601 fixed classes |
| `yolov8n-oiv7.names.json` | 12 KB | — | class index → name, 601 entries |
| `yolov8s-worldv2-danmu.onnx` | 50.4 MB | `1x48x8400` | YOLO-World small, 44 furniture prompts baked in |

Both are opset 12, input `1x3x640x640` NCHW, and share the standard YOLOv8
detect-head layout: channels are `cx, cy, w, h` then per-class scores
(`605 - 4 = 601`, `48 - 4 = 44`) across 8400 anchors. **No objectness channel and
no sigmoid** — scores are ready to threshold. Boxes are in letterboxed 640-space
and must be unpadded.

Both use only standard ONNX ops (no custom domains), so they run under
`onnxruntime-web` on WASM or WebGPU.

## The baked vocabulary

`yolov8s-worldv2-danmu.onnx` was produced with Ultralytics `set_classes()`,
which runs the CLIP text encoder once at export and freezes the embeddings into
the graph — so no text encoder is needed at runtime. **Class index N is
`PROMPTS[N]`**, in exactly this order:

```js
const PROMPTS = [
  'sofa', 'couch', 'armchair',
  'chair', 'office chair', 'stool',
  'table', 'coffee table', 'dining table',
  'desk',
  'bed', 'mattress',
  'nightstand',
  'wardrobe', 'closet', 'chest of drawers', 'storage cabinet',
  'shelf', 'bookshelf', 'shoe rack',
  'mirror',
  'curtain', 'window curtain', 'window blind',
  'picture frame', 'wall art', 'poster',
  'lamp', 'light bulb', 'ceiling light',
  'ceiling fan', 'electric fan',
  'refrigerator',
  'potted plant',
  'door', 'wooden door',
  'computer monitor',
  'television',
  'window',
  'laptop',
  'washing machine',
  'microwave oven',
  'clothes rack', 'hanging clothes',
];
```

Do not reorder that list — it is the model's channel order, and an off-by-one
mislabels every detection silently.

Several phrases deliberately collapse to one concept (`clothes rack`, `hanging
clothes`, `storage cabinet` → wardrobe). Real rooms contain clothes rails and
stacked fabric cubes, not the canonical `Wardrobe` a fixed-label model was
trained on, and naming what is actually there is the whole advantage of an
open-vocabulary model. To retarget it, re-export with your own prompt list.

## Loading

```js
const BASE = 'https://huggingface.co/DearthAI/danmu-detector/resolve/main/';

const providers = [];
if (typeof navigator !== 'undefined' && 'gpu' in navigator) providers.push('webgpu');
providers.push('wasm');

const oiv   = await ort.InferenceSession.create(BASE + 'yolov8n-oiv7.onnx', { executionProviders: providers });
const world = await ort.InferenceSession.create(BASE + 'yolov8s-worldv2-danmu.onnx', { executionProviders: providers });
const names = await (await fetch(BASE + 'yolov8n-oiv7.names.json')).json();
```

Use the `/resolve/` URL, not `/blob/` — `/blob/` returns the HTML viewer page,
which passes a HEAD status check and then hands your runtime a page of markup.

## Preprocessing

Letterbox to 640×640 preserving aspect ratio, pad with grey, normalize to `0..1`,
feed **planar** RGB (all R, then all G, then all B — not interleaved).

```js
const INPUT = 640;

function toTensor(img, ox, oy, cw, ch) {      // crop rect in source pixels
  const scale = INPUT / Math.max(cw, ch);
  const sw = Math.round(cw * scale), sh = Math.round(ch * scale);
  const dw = Math.floor((INPUT - sw) / 2), dh = Math.floor((INPUT - sh) / 2);

  const c = document.createElement('canvas');
  c.width = c.height = INPUT;
  const ctx = c.getContext('2d');
  ctx.fillStyle = '#727272';                  // letterbox grey
  ctx.fillRect(0, 0, INPUT, INPUT);
  ctx.drawImage(img, ox, oy, cw, ch, dw, dh, sw, sh);

  const px = ctx.getImageData(0, 0, INPUT, INPUT).data;
  const area = INPUT * INPUT;
  const data = new Float32Array(3 * area);
  for (let i = 0; i < area; i++) {
    data[i]            = px[i * 4]     / 255;
    data[area + i]     = px[i * 4 + 1] / 255;
    data[2 * area + i] = px[i * 4 + 2] / 255;
  }
  return { data, scale, dw, dh };
}
```

## Tile — it matters more than model size

Letterboxing a 2000px wide-angle photo to 640 shrinks mid-sized furniture below
what these models resolve. Running the whole frame **plus 2×2 tiles at 15%
overlap** took nano from 4/19 to 7/19 for zero extra download, and the world
model from 7/19 to 10/19. Overlap keeps objects straddling a seam intact.

```js
function tilesFor(iw, ih) {
  const ox = iw * 0.15, oy = ih * 0.15;
  const crops = [{ ox: 0, oy: 0, cw: iw, ch: ih }];
  for (let r = 0; r < 2; r++)
    for (let c = 0; c < 2; c++) {
      const x0 = Math.max(0, c * iw / 2 - ox), y0 = Math.max(0, r * ih / 2 - oy);
      const x1 = Math.min(iw, (c + 1) * iw / 2 + ox);
      const y1 = Math.min(ih, (r + 1) * ih / 2 + oy);
      crops.push({ ox: x0, oy: y0, cw: x1 - x0, ch: y1 - y0 });
    }
  return crops;
}
```

## Decoding, and merging the two models

Collect candidates from **both models over every tile into one pool in
normalized whole-image coordinates**, then run a single NMS. That one pass
resolves per-tile duplicates, cross-tile seam duplicates, and the same object
found by both models.

```js
const CONF = 0.35, IOU_T = 0.45, MAX_PER_IMAGE = 12;
const pool = [];

for (const crop of tilesFor(iw, ih)) {
  const pre = toTensor(img, crop.ox, crop.oy, crop.cw, crop.ch);
  const toX = v => (crop.ox + (v - pre.dw) / pre.scale) / iw;
  const toY = v => (crop.oy + (v - pre.dh) / pre.scale) / ih;

  for (const [sess, labelOf] of [
    [oiv,   i => names[String(i)]],
    [world, i => PROMPTS[i]],
  ]) {
    const t = new ort.Tensor('float32', pre.data.slice(), [1, 3, INPUT, INPUT]);
    const out = (await sess.run({ [sess.inputNames[0]]: t }))[sess.outputNames[0]];
    const [, channels, anchors] = out.dims;
    const nc = channels - 4, d = out.data;
    const at = (ch, a) => d[ch * anchors + a];

    for (let a = 0; a < anchors; a++) {
      let best = 0, bestC = -1;
      for (let ci = 0; ci < nc; ci++) {
        const s = at(4 + ci, a);
        if (s > best) { best = s; bestC = ci; }
      }
      if (best < CONF || bestC < 0) continue;
      pool.push({
        x: toX(at(0, a)), y: toY(at(1, a)),
        w: at(2, a) / pre.scale / iw,
        h: at(3, a) / pre.scale / ih,
        conf: best, label: labelOf(bestC),
      });
    }
  }
}
```

Then NMS in that same normalized space, and convert centre-form to top-left:

```js
function iou(a, b) {
  const x1 = Math.max(a.x - a.w / 2, b.x - b.w / 2);
  const y1 = Math.max(a.y - a.h / 2, b.y - b.h / 2);
  const x2 = Math.min(a.x + a.w / 2, b.x + b.w / 2);
  const y2 = Math.min(a.y + a.h / 2, b.y + b.h / 2);
  const inter = Math.max(0, x2 - x1) * Math.max(0, y2 - y1);
  return inter / (a.w * a.h + b.w * b.h - inter + 1e-9);
}

const kept = [];
for (const b of [...pool].sort((p, q) => q.conf - p.conf)) {
  if (kept.every(k => iou(k, b) < IOU_T)) kept.push(b);
  if (kept.length >= MAX_PER_IMAGE) break;
}
const boxes = kept.map(b => ({
  label: b.label, conf: b.conf,
  x: b.x - b.w / 2, y: b.y - b.h / 2, w: b.w, h: b.h,
}));
```

Subtracting `dw`/`dh` before dividing is the step that is easy to miss — skip it
and every box drifts toward the image centre on non-square inputs.

## Tuning notes

Measured, so you don't have to rediscover them:

- **Confidence 0.35.** Dropping to 0.20 gained one object and added a spurious
  `sofa(0.29)`.
- **`MAX_PER_IMAGE` 12.** Raising it to 30 added one box and changed no score.
- **Bigger is not better.** `yolov8s/m/x-oiv7` (46 / 105 / 275 MB) all score the
  same 7/19 as the 14 MB nano. Spend the bytes on the second model instead.
- **Still missed** at 13/19: doors in some framings, wall art, and curtains in a
  frame that is almost entirely curtain. Treat the output as a head start on
  manual correction, not a finished result.

## Reproducing

```bash
pip install --index-url https://download.pytorch.org/whl/cpu torch
pip install ultralytics onnx onnxslim onnxruntime clip-anytorch ftfy

python - <<'PY'
from ultralytics import YOLO
YOLO('yolov8n-oiv7.pt').export(format='onnx', imgsz=640, opset=12)

m = YOLO('yolov8s-worldv2.pt')
m.set_classes(PROMPTS)          # the 44 phrases above, in order
m.export(format='onnx', imgsz=640, opset=12)
PY
```

CPU-only torch is enough — export needs no GPU, and it keeps the install near
200 MB instead of 2.5 GB.

## Licence

**AGPL-3.0.** Both are Ultralytics YOLO weights, licensed AGPL-3.0 by
[Ultralytics](https://github.com/ultralytics/ultralytics). The AGPL obligations
attach to these artifacts and travel with them.

They are hosted here, separately from the application that consumes them,
precisely so the licences stay distinct: that app is MIT, and it fetches these
weights at runtime rather than bundling or redistributing them. If you vendor
these files into your own project, AGPL-3.0 applies to you — a permissive
licence on the surrounding code does not cover them.