Spaces:
Running on Zero
Running on Zero
Upload 44 files
Browse files- DEPLOY.md +13 -0
- README.md +36 -13
- app.py +130 -40
- requirements.txt +11 -1
- src/real_vision.py +341 -0
DEPLOY.md
CHANGED
|
@@ -128,3 +128,16 @@ branding needs to change.
|
|
| 128 |
**Dependency note:** `requirements.txt` pins `pydantic==2.10.6`. Gradio 5.9.1's API-schema generation
|
| 129 |
crashes on newer pydantic releases (`TypeError: argument of type 'bool' is not iterable`), which would
|
| 130 |
make the Space fail to start after a rebuild. Don't remove the pin unless you also upgrade Gradio.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 128 |
**Dependency note:** `requirements.txt` pins `pydantic==2.10.6`. Gradio 5.9.1's API-schema generation
|
| 129 |
crashes on newer pydantic releases (`TypeError: argument of type 'bool' is not iterable`), which would
|
| 130 |
make the Space fail to start after a rebuild. Don't remove the pin unless you also upgrade Gradio.
|
| 131 |
+
|
| 132 |
+
## Visual Inventory Scan (real photos): deployment notes
|
| 133 |
+
|
| 134 |
+
- `requirements.txt` adds `ultralytics`, `opencv-python`, and Ultralytics' `CLIP` package (from GitHub).
|
| 135 |
+
PyTorch is already present on Hugging Face Spaces images. I could not run a Space build from my
|
| 136 |
+
environment, so **check the build log after pushing**.
|
| 137 |
+
- **If that build fails**, delete the last block of `requirements.txt` (`ultralytics`, `ftfy`, `regex`,
|
| 138 |
+
`git+https://github.com/ultralytics/CLIP.git`) but keep `opencv-python`. I tested this degraded mode:
|
| 139 |
+
the whole app still works, and the real-photo tab explains that the detector isn't available.
|
| 140 |
+
- Memory: open-vocabulary mode keeps a ~600 MB text encoder plus the model in RAM; fine on the free
|
| 141 |
+
16 GB CPU tier. The first request in that mode takes ~1–2 minutes (weight download).
|
| 142 |
+
- The app calls `device="cpu"` explicitly, so it also runs on ZeroGPU hardware without a GPU.
|
| 143 |
+
- Licence: Ultralytics is AGPL-3.0 — see README.
|
README.md
CHANGED
|
@@ -7,7 +7,7 @@ sdk: gradio
|
|
| 7 |
sdk_version: 5.9.1
|
| 8 |
app_file: app.py
|
| 9 |
pinned: false
|
| 10 |
-
license:
|
| 11 |
short_description: LLM warehouse copilot, predictive maintenance & CV
|
| 12 |
---
|
| 13 |
|
|
@@ -33,8 +33,13 @@ metrics on held-out test data.
|
|
| 33 |
3. **⚠️ Predictive Maintenance** — Isolation Forest anomaly detector (calibrated threshold). Single
|
| 34 |
readings via sliders, **CSV batch scoring** (sample file provided), and optional **Slack alerts**
|
| 35 |
(real if `SLACK_WEBHOOK_URL` is set, otherwise a clearly-labelled preview — never claimed as sent).
|
| 36 |
-
4. **📷 Visual Inventory Scan** —
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
5. **📊 Model Evaluation** — graphical, held-out metrics for every component, including the vision
|
| 39 |
pipeline's accuracy-vs-noise curve.
|
| 40 |
6. **ℹ️ About** — dataset provenance, model function table, architecture, how to test.
|
|
@@ -86,6 +91,7 @@ smart-warehouse-ai/
|
|
| 86 |
├── requirements.txt
|
| 87 |
├── src/
|
| 88 |
│ ├── vision_model.py # classical-CV rack scanner (+ synthetic scene generator)
|
|
|
|
| 89 |
│ ├── alerting.py # Slack alert (real webhook or preview) + email preview
|
| 90 |
│ ├── data_generation.py # synthetic intent / inventory / sensor datasets
|
| 91 |
│ ├── knowledge_base.py # warehouse-ops knowledge base (RAG source docs)
|
|
@@ -114,7 +120,8 @@ benchmark). Headline numbers from the included run:
|
|
| 114 |
| Anomaly detector | F1 | ~98% |
|
| 115 |
| Anomaly detector | ROC-AUC | ~1.00 |
|
| 116 |
| RAG retriever | Hit-rate@1 / @2 | 88% / 100% |
|
| 117 |
-
|
|
|
|
|
| 118 |
|
| 119 |
*(Computed on synthetic, held-out test data — see the Evaluation tab for
|
| 120 |
methodology notes.)*
|
|
@@ -146,15 +153,31 @@ number: it's threshold-independent by construction, so it confirms the
|
|
| 146 |
model's *ranking* of anomalies was always excellent — the entire story here
|
| 147 |
is about honestly picking the cutoff, not about detection ability.
|
| 148 |
|
| 149 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 150 |
|
| 151 |
-
|
| 152 |
|
| 153 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
|
| 155 |
-
The
|
| 156 |
-
|
| 157 |
-
(
|
| 158 |
-
|
| 159 |
-
padded pixel area to *occupied grid cells* to match how ground truth is defined. All scenes are synthetic —
|
| 160 |
-
a real deployment would fine-tune a detector (e.g. YOLO) on real photos.
|
|
|
|
| 7 |
sdk_version: 5.9.1
|
| 8 |
app_file: app.py
|
| 9 |
pinned: false
|
| 10 |
+
license: agpl-3.0
|
| 11 |
short_description: LLM warehouse copilot, predictive maintenance & CV
|
| 12 |
---
|
| 13 |
|
|
|
|
| 33 |
3. **⚠️ Predictive Maintenance** — Isolation Forest anomaly detector (calibrated threshold). Single
|
| 34 |
readings via sliders, **CSV batch scoring** (sample file provided), and optional **Slack alerts**
|
| 35 |
(real if `SLACK_WEBHOOK_URL` is set, otherwise a clearly-labelled preview — never claimed as sent).
|
| 36 |
+
4. **📷 Visual Inventory Scan** — two sub-tabs:
|
| 37 |
+
- **Real photos (YOLO):** upload a warehouse photo (or load an openly-licensed Wikimedia Commons
|
| 38 |
+
example, fetched at runtime with author/licence shown). *Standard* mode = COCO-pretrained YOLOv8
|
| 39 |
+
(finds people/vehicles; COCO has **no** pallet/forklift/box class). *Find by name* mode =
|
| 40 |
+
open-vocabulary **YOLOE** — type "pallet, cardboard box, forklift…" and it searches zero-shot.
|
| 41 |
+
- **Synthetic rack (classical-CV baseline):** the original OpenCV pipeline on synthetic racks with
|
| 42 |
+
known ground truth, so accuracy *can* be measured.
|
| 43 |
5. **📊 Model Evaluation** — graphical, held-out metrics for every component, including the vision
|
| 44 |
pipeline's accuracy-vs-noise curve.
|
| 45 |
6. **ℹ️ About** — dataset provenance, model function table, architecture, how to test.
|
|
|
|
| 91 |
├── requirements.txt
|
| 92 |
├── src/
|
| 93 |
│ ├── vision_model.py # classical-CV rack scanner (+ synthetic scene generator)
|
| 94 |
+
│ ├── real_vision.py # real-photo detection: YOLOv8 (COCO) + YOLOE (open-vocabulary)
|
| 95 |
│ ├── alerting.py # Slack alert (real webhook or preview) + email preview
|
| 96 |
│ ├── data_generation.py # synthetic intent / inventory / sensor datasets
|
| 97 |
│ ├── knowledge_base.py # warehouse-ops knowledge base (RAG source docs)
|
|
|
|
| 120 |
| Anomaly detector | F1 | ~98% |
|
| 121 |
| Anomaly detector | ROC-AUC | ~1.00 |
|
| 122 |
| RAG retriever | Hit-rate@1 / @2 | 88% / 100% |
|
| 123 |
+
| Synthetic-rack CV baseline (noise 0.4) | Exact box-count match | ~88% (100% clean → ~77% at max noise) |
|
| 124 |
+
| Real-photo YOLO / YOLOE | — | **Not benchmarked** (no labelled warehouse photos available) |
|
| 125 |
|
| 126 |
*(Computed on synthetic, held-out test data — see the Evaluation tab for
|
| 127 |
methodology notes.)*
|
|
|
|
| 153 |
model's *ranking* of anomalies was always excellent — the entire story here
|
| 154 |
is about honestly picking the cutoff, not about detection ability.
|
| 155 |
|
| 156 |
+
## Real-photo detection: what was and wasn't verified
|
| 157 |
+
|
| 158 |
+
| | Status |
|
| 159 |
+
|---|---|
|
| 160 |
+
| YOLOv8n / YOLOv8s (COCO) on a real photo | ✅ run end-to-end; detections visually checked |
|
| 161 |
+
| YOLOE open-vocabulary on a real photo | ✅ run end-to-end through the app's API; one duplicate-box over-count observed, hence the user-facing IoU slider |
|
| 162 |
+
| Accuracy on **warehouse** photos | ❌ **Not measured** — no labelled warehouse images were available |
|
| 163 |
+
| Wikimedia example photos | ⚠️ Fetched at runtime on the Space; my development sandbox couldn't reach Wikimedia, so these were *not* run through the detector before delivery (the failure path *was* tested) |
|
| 164 |
+
| YOLO-World / OWLv2 / Grounding DINO | Not shipped — their weights are hosted where I couldn't verify them |
|
| 165 |
+
|
| 166 |
+
Photos are processed in memory and never stored; Ultralytics' anonymous usage sync is switched off.
|
| 167 |
+
First use downloads weights from the official Ultralytics GitHub releases (≈630 MB for open-vocabulary
|
| 168 |
+
mode, once per container).
|
| 169 |
|
| 170 |
+
## Image attributions
|
| 171 |
|
| 172 |
+
Example photos are **not bundled**; they're fetched from Wikimedia Commons when selected, and shown with
|
| 173 |
+
author and licence:
|
| 174 |
+
- *Modern warehouse with pallet rack storage system.jpg* — Axisadman, CC BY-SA 3.0
|
| 175 |
+
- *Allentown Project 042.jpg* — Chrisking, CC BY-SA 3.0
|
| 176 |
+
- *576 CHEP pallets.jpg* — train_photos (Flickr), CC BY-SA 2.0
|
| 177 |
+
|
| 178 |
+
## License
|
| 179 |
|
| 180 |
+
**AGPL-3.0.** The real-photo detector uses Ultralytics YOLO, whose code and pretrained weights are
|
| 181 |
+
AGPL-3.0; serving them as a network app means the combined project is distributed under the same terms
|
| 182 |
+
(source is available in this repository). If you want to reuse the project in a closed-source product,
|
| 183 |
+
Ultralytics sells a commercial licence, or you can swap in a differently-licensed detector.
|
|
|
|
|
|
app.py
CHANGED
|
@@ -29,6 +29,10 @@ from src.anomaly_model import load_artifacts as load_anomaly_artifacts
|
|
| 29 |
from src.anomaly_model import score_reading
|
| 30 |
from src.alerting import send_anomaly_alert
|
| 31 |
from src.vision_model import generate_rack_scene, detect_boxes
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
from src.data_generation import generate_inventory_db, generate_orders_db
|
| 33 |
from src.intent_model import INTENT_DESCRIPTIONS, load_pipeline, predict as intent_predict
|
| 34 |
from src.inventory_db import query_inventory, query_orders
|
|
@@ -319,6 +323,30 @@ def vision_scan_fn(fill_density, noise_level, seed_state):
|
|
| 319 |
return scene.image, result.annotated_image, summary, seed + 1
|
| 320 |
|
| 321 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 322 |
# ==========================================================================
|
| 323 |
# TAB 4 -- Model Evaluation
|
| 324 |
# ==========================================================================
|
|
@@ -512,7 +540,19 @@ model by default.
|
|
| 512 |
## Extended capabilities
|
| 513 |
|
| 514 |
- **Multi-turn conversation** — the assistant keeps the last few turns, so follow-ups like *"what about Zone D instead?"* work.
|
| 515 |
-
- **📷 Visual Inventory Scan** —
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 516 |
- **Batch CSV scoring** — upload a CSV of sensor readings (a sample is provided) to score a whole shift's logs at once, and download the results.
|
| 517 |
- **Slack alerting** — anomalies can trigger a Slack alert. Real if a `SLACK_WEBHOOK_URL` secret is set; otherwise a clearly-labelled preview that is *never* presented as sent.
|
| 518 |
- **Bilingual UI** — switch the explanatory text between English and 日本語 with the selector at the top. (Charts and metric tables stay in English; the assistant answers in whatever language you ask in.)
|
|
@@ -622,21 +662,23 @@ I18N = {
|
|
| 622 |
"異常時はSlack通知が可能です(<code>SLACK_WEBHOOK_URL</code>設定時は実送信、未設定時はプレビュー表示)。",
|
| 623 |
},
|
| 624 |
"badge_vision": {
|
| 625 |
-
"en": "🧪 <b>Source:</b>
|
| 626 |
-
"
|
| 627 |
-
"<b>
|
| 628 |
-
"
|
| 629 |
-
"
|
| 630 |
-
"
|
| 631 |
-
"
|
| 632 |
-
"
|
| 633 |
-
|
| 634 |
-
|
| 635 |
-
"
|
| 636 |
-
"
|
| 637 |
-
"<b>
|
| 638 |
-
"
|
| 639 |
-
"
|
|
|
|
|
|
|
| 640 |
},
|
| 641 |
"badge_eval": {
|
| 642 |
"en": "🧪 All charts below are computed on <b>held-out test data</b> by <code>build_artifacts.py</code> — fully reproducible, not cherry-picked.",
|
|
@@ -663,7 +705,7 @@ AS/RS、コンベヤ・仕分け、AGV/AMR、WMS/WCSといった倉庫自動化
|
|
| 663 |
| **意図分類**による問い合わせの振り分け(8カテゴリ) | AI Assistant / Inventory | TF-IDF + ロジスティック回帰 |
|
| 664 |
| 在庫・注文の**自然文照会** | Inventory & Order Query | 正規表現によるスロット抽出 + 意図分類 |
|
| 665 |
| センサーデータの**予知保全**(1件判定/CSV一括判定/Slack通知) | Predictive Maintenance | Isolation Forest(教師なし)+ 較正済みしきい値 |
|
| 666 |
-
| ラック
|
| 667 |
| 全コンポーネントの**再現可能な評価** | Model Evaluation | scikit-learn指標、ホールドアウト検証 |
|
| 668 |
|
| 669 |
## データセットの概要
|
|
@@ -690,26 +732,26 @@ AS/RS、コンベヤ・仕分け、AGV/AMR、WMS/WCSといった倉庫自動化
|
|
| 690 |
| LLMアシスタント | 根拠に基づく自然な回答を生成 | ホスト型LLM(複数モデルのフォールバック) | 質問 + 根拠 + 会話履歴 → 回答 |
|
| 691 |
| 在庫・注文照会 | 質問をテーブル検索に変換 | 正規表現 + 意図分類 | 文章 → 絞り込み結果 |
|
| 692 |
| 異常検知 | 設備の異常な計測値を早期に検出 | Isolation Forest + 較正済みしきい値 | 4つのセンサー値 → 正常/異常 + スコア |
|
| 693 |
-
|
|
| 694 |
|
| 695 |
## 試し方
|
| 696 |
|
| 697 |
1. **AI Assistant** — 例の質問を選ぶか自由に入力します。続けて「Zone Dの場合は?」などと尋ねると、直前の文脈を踏まえて回答します。
|
| 698 |
2. **Inventory & Order Query** — 「SKU-1042 は Zone B に何個ある?」「注文 #10007 の状況は?」などを試します。
|
| 699 |
3. **Predictive Maintenance** — プリセットまたはスライダーで1件を判定します。「CSV一括判定」でサンプルCSVをダウンロードし、そのままアップロードして一括判定も試せます。
|
| 700 |
-
4. **Visual Inventory Scan** —
|
| 701 |
5. **Model Evaluation** — すべての図表はテストデータで算出しており、`build_artifacts.py`を再実行すれば同じ数値を再現できます。
|
| 702 |
|
| 703 |
## 評価の信頼性について
|
| 704 |
|
| 705 |
開発中に2件のデータリーク(意図分類での学習/テスト間の文面重複、異常検知でのしきい値への正解率の混入)を発見・修正しました。
|
| 706 |
修正後の指標は、意図分類 正解率 {intent_eval.get('accuracy', 0):.1%}、異常検知 F1 {anomaly_eval.get('f1', 0):.1%} です。
|
| 707 |
-
画像
|
| 708 |
|
| 709 |
## 制約と今後の課題
|
| 710 |
|
| 711 |
- データはすべて**合成**です。実運用ではWMS/WCSのAPIや実際のセンサーログに接続します。
|
| 712 |
-
-
|
| 713 |
- Slack通知は`SLACK_WEBHOOK_URL`未設定の場合はプレビューのみで、実際には送信されません。
|
| 714 |
"""
|
| 715 |
|
|
@@ -825,24 +867,64 @@ with gr.Blocks(theme=gr.themes.Soft(primary_hue="blue"), css=CUSTOM_CSS, title="
|
|
| 825 |
with gr.Tab("📷 Visual Inventory Scan"):
|
| 826 |
with gr.Accordion("🧪 About this data & function / データ・機能について", open=False):
|
| 827 |
badge_vision = gr.HTML(data_badge(I18N["badge_vision"]["en"]))
|
| 828 |
-
gr.
|
| 829 |
-
"
|
| 830 |
-
|
| 831 |
-
|
| 832 |
-
|
| 833 |
-
|
| 834 |
-
|
| 835 |
-
|
| 836 |
-
|
| 837 |
-
|
| 838 |
-
|
| 839 |
-
|
| 840 |
-
|
| 841 |
-
|
| 842 |
-
|
| 843 |
-
|
| 844 |
-
|
| 845 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 846 |
|
| 847 |
with gr.Tab("📊 Model Evaluation"):
|
| 848 |
with gr.Accordion("🧪 About this data & function / データ・機能について", open=False):
|
|
@@ -873,7 +955,7 @@ with gr.Blocks(theme=gr.themes.Soft(primary_hue="blue"), css=CUSTOM_CSS, title="
|
|
| 873 |
gr.Markdown("### 4. Latency Benchmark (CPU, local components)")
|
| 874 |
gr.Image(os.path.join(ASSETS_DIR, "latency_bar.png"), show_label=False, container=False)
|
| 875 |
|
| 876 |
-
gr.Markdown("### 5. Visual Inventory Scan — exact box-count match
|
| 877 |
vision_eval.get("exact_match_rate", 0), vision_eval.get("standard_noise_level", "?"), vision_eval.get("count_mae", 0)
|
| 878 |
))
|
| 879 |
with gr.Row():
|
|
@@ -882,6 +964,14 @@ with gr.Blocks(theme=gr.themes.Soft(primary_hue="blue"), css=CUSTOM_CSS, title="
|
|
| 882 |
with gr.Row():
|
| 883 |
gr.Image(os.path.join(ASSETS_DIR, "vision_sample_raw.png"), label="Sample input", container=True)
|
| 884 |
gr.Image(os.path.join(ASSETS_DIR, "vision_sample_annotated.png"), label="Sample detection", container=True)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 885 |
|
| 886 |
with gr.Accordion("📋 Full metrics tables & methodology notes", open=False):
|
| 887 |
gr.Markdown(eval_intent_section_md())
|
|
|
|
| 29 |
from src.anomaly_model import score_reading
|
| 30 |
from src.alerting import send_anomaly_alert
|
| 31 |
from src.vision_model import generate_rack_scene, detect_boxes
|
| 32 |
+
from src.real_vision import (
|
| 33 |
+
DEFAULT_PROMPTS, EXAMPLE_PHOTOS, MODEL_CHOICES, detect_real, fetch_example,
|
| 34 |
+
summarize as summarize_real, vision_stack_available,
|
| 35 |
+
)
|
| 36 |
from src.data_generation import generate_inventory_db, generate_orders_db
|
| 37 |
from src.intent_model import INTENT_DESCRIPTIONS, load_pipeline, predict as intent_predict
|
| 38 |
from src.inventory_db import query_inventory, query_orders
|
|
|
|
| 323 |
return scene.image, result.annotated_image, summary, seed + 1
|
| 324 |
|
| 325 |
|
| 326 |
+
def real_detect_fn(image, mode, model_choice, prompts_text, conf, iou=0.7):
|
| 327 |
+
"""Run a real-photo detector (YOLO) on an uploaded / example image."""
|
| 328 |
+
if image is None:
|
| 329 |
+
return None, "⬆️ Upload a photo, or load an example, first."
|
| 330 |
+
ok, why = vision_stack_available()
|
| 331 |
+
if not ok:
|
| 332 |
+
return None, (f"⚠️ The detector stack (ultralytics + PyTorch) isn't available on this server "
|
| 333 |
+
f"(`{why}`). The rest of the app is unaffected.")
|
| 334 |
+
try:
|
| 335 |
+
res = detect_real(image, mode=mode, model_choice=model_choice, prompts_text=prompts_text, conf=conf, iou=iou)
|
| 336 |
+
except ValueError as e: # bad user input (e.g. empty prompt box) -> plain guidance
|
| 337 |
+
return None, f"✏️ {e}"
|
| 338 |
+
except Exception as e: # noqa: BLE001
|
| 339 |
+
return None, (f"⚠️ Detection failed: `{type(e).__name__}: {e}`\n\n"
|
| 340 |
+
f"The first run downloads model weights from GitHub, so a network problem there "
|
| 341 |
+
f"is the most common cause — try again, or switch detector mode.")
|
| 342 |
+
return res.annotated, summarize_real(res)
|
| 343 |
+
|
| 344 |
+
|
| 345 |
+
def load_example_fn(name):
|
| 346 |
+
img, msg = fetch_example(name)
|
| 347 |
+
return img, msg
|
| 348 |
+
|
| 349 |
+
|
| 350 |
# ==========================================================================
|
| 351 |
# TAB 4 -- Model Evaluation
|
| 352 |
# ==========================================================================
|
|
|
|
| 540 |
## Extended capabilities
|
| 541 |
|
| 542 |
- **Multi-turn conversation** — the assistant keeps the last few turns, so follow-ups like *"what about Zone D instead?"* work.
|
| 543 |
+
- **📷 Visual Inventory Scan** — runs YOLO on **real warehouse photos** (upload your own, or load an openly-licensed example). *Standard* mode (COCO-pretrained YOLOv8) finds people and vehicles but has no pallet/forklift/box class; *Find by name* mode (open-vocabulary YOLOE) searches for whatever you type, zero-shot. **Honest limit:** accuracy on real warehouse photos is *not measured* (no labelled warehouse images were available). A classical-CV baseline on synthetic racks, where ground truth is known, *is* measured: 100% on clean renders, degrading under simulated noise (see Model Evaluation).
|
| 544 |
+
|
| 545 |
+
**Detector options considered for real photos** (latency = rough figures from a development CPU, after model load):
|
| 546 |
+
|
| 547 |
+
| Option | Finds pallets / forklifts / boxes? | Weight & speed | Status |
|
| 548 |
+
|---|---|---|---|
|
| 549 |
+
| YOLOv8 (COCO) | **No** — no such classes; finds people & vehicles | 6–22 MB, ~75 ms | ✅ Shipped; run on a real photo |
|
| 550 |
+
| YOLOE (open-vocabulary) | Yes, zero-shot via text prompts | 28 MB + ~600 MB text encoder, ~250 ms; first load ≈ 1–2 min | ✅ Shipped; run on a real photo (not on warehouse photos) |
|
| 551 |
+
| YOLO-World (open-vocabulary) | Yes, zero-shot | text encoder hosted on a CDN I couldn't reach | ❌ Not shipped — couldn't verify it |
|
| 552 |
+
| OWLv2 / Grounding DINO | Yes, zero-shot, often stronger | hundreds of MB, seconds per image on CPU; weights on the Hugging Face Hub, unreachable from my environment | ❌ Not shipped — couldn't verify it |
|
| 553 |
+
| YOLO fine-tuned on a warehouse dataset | Best accuracy | needs licensed, labelled warehouse photos + training | ⏳ Recommended next step |
|
| 554 |
+
| Classical CV (OpenCV) | Only clean synthetic racks | tiny | ✅ Kept as a *measured* baseline |
|
| 555 |
+
|
| 556 |
- **Batch CSV scoring** — upload a CSV of sensor readings (a sample is provided) to score a whole shift's logs at once, and download the results.
|
| 557 |
- **Slack alerting** — anomalies can trigger a Slack alert. Real if a `SLACK_WEBHOOK_URL` secret is set; otherwise a clearly-labelled preview that is *never* presented as sent.
|
| 558 |
- **Bilingual UI** — switch the explanatory text between English and 日本語 with the selector at the top. (Charts and metric tables stay in English; the assistant answers in whatever language you ask in.)
|
|
|
|
| 662 |
"異常時はSlack通知が可能です(<code>SLACK_WEBHOOK_URL</code>設定時は実送信、未設定時はプレビュー表示)。",
|
| 663 |
},
|
| 664 |
"badge_vision": {
|
| 665 |
+
"en": "🧪 <b>Source:</b> <b>Real photos:</b> anything you upload (processed in memory, not stored), plus three openly-licensed "
|
| 666 |
+
"example photos fetched at runtime from Wikimedia Commons with author and license shown — no third-party images are bundled "
|
| 667 |
+
"in this repo. <b>Synthetic baseline:</b> rack images generated by <code>src/vision_model.py</code>, where ground truth is known.<br>"
|
| 668 |
+
"<b>What it describes:</b> real warehouse scenes (racking, pallets, forklifts, people) for the YOLO detectors; a simplified 3×6 storage rack "
|
| 669 |
+
"with box shapes for the baseline.<br>"
|
| 670 |
+
"<b>Function:</b> <i>Standard</i> mode runs COCO-pretrained YOLOv8, which finds people and vehicles but has <b>no pallet, forklift or box "
|
| 671 |
+
"class</b>. <i>Find by name</i> mode runs open-vocabulary YOLOE: type what to look for and it searches zero-shot. "
|
| 672 |
+
"<b>Accuracy on real warehouse photos has not been measured</b> — no labelled warehouse images were available — so treat counts as a "
|
| 673 |
+
"starting point. The classical-CV baseline on synthetic racks <i>is</i> measured (see Model Evaluation).",
|
| 674 |
+
"ja": "🧪 <b>データの出典:</b><b>実写真:</b>アップロードした画像(メモリ上で処理し、保存しません)と、Wikimedia Commonsから実行時に取得する"
|
| 675 |
+
"オープンライセンスの例示写真3枚(作者・ライセンスを表示)。第三者の画像はリポジトリに同梱していません。"
|
| 676 |
+
"<b>合成ベースライン:</b><code>src/vision_model.py</code>で生成した、正解が既知のラック画像。<br>"
|
| 677 |
+
"<b>内容:</b>YOLO検出器向けの実際の倉庫シーン(ラック、パレット、フォークリフト、作業者)と、ベースライン用の簡易な3×6ラック画像。<br>"
|
| 678 |
+
"<b>機能:</b><i>標準</i>モードはCOCO学習済みYOLOv8を使用し、人と車両は検出できますが、<b>パレット・フォークリフト・箱のクラスはありません</b>。"
|
| 679 |
+
"<i>名前で検索</i>モードはオープン語彙のYOLOEで、探したい物を文字で入力してゼロショット検出します。"
|
| 680 |
+
"<b>実際の倉庫写真での精度は未測定</b>です(ラベル付き倉庫画像が入手できなかったため)。件数は目安としてご利用ください。"
|
| 681 |
+
"合成ラックでの古典的画像処理ベースラインは測定済みです(Model Evaluationタブ参照)。",
|
| 682 |
},
|
| 683 |
"badge_eval": {
|
| 684 |
"en": "🧪 All charts below are computed on <b>held-out test data</b> by <code>build_artifacts.py</code> — fully reproducible, not cherry-picked.",
|
|
|
|
| 705 |
| **意図分類**による問い合わせの振り分け(8カテゴリ) | AI Assistant / Inventory | TF-IDF + ロジスティック回帰 |
|
| 706 |
| 在庫・注文の**自然文照会** | Inventory & Order Query | 正規表現によるスロット抽出 + 意図分類 |
|
| 707 |
| センサーデータの**予知保全**(1件判定/CSV一括判定/Slack通知) | Predictive Maintenance | Isolation Forest(教師なし)+ 較正済みしきい値 |
|
| 708 |
+
| **実写真の物体検出**(人・車両/名前指定でパレット等)+合成ラックのベースライン | Visual Inventory Scan | YOLOv8 / YOLOE + 古典的画像処理 |
|
| 709 |
| 全コンポーネントの**再現可能な評価** | Model Evaluation | scikit-learn指標、ホールドアウト検証 |
|
| 710 |
|
| 711 |
## データセットの概要
|
|
|
|
| 732 |
| LLMアシスタント | 根拠に基づく自然な回答を生成 | ホスト型LLM(複数モデルのフォールバック) | 質問 + 根拠 + 会話履歴 → 回答 |
|
| 733 |
| 在庫・注文照会 | 質問をテーブル検索に変換 | 正規表現 + 意図分類 | 文章 → 絞り込み結果 |
|
| 734 |
| 異常検知 | 設備の異常な計測値を早期に検出 | Isolation Forest + 較正済みしきい値 | 4つのセンサー値 → 正常/異常 + スコア |
|
| 735 |
+
| 画像検出器 | 実写真から物体を検出 | YOLOv8(COCO)/YOLOE(名前指定)、ベースラインは古典的画像処理 | 画像 → 検出枠 + クラス別件数 |
|
| 736 |
|
| 737 |
## 試し方
|
| 738 |
|
| 739 |
1. **AI Assistant** — 例の質問を選ぶか自由に入力します。続けて「Zone Dの場合は?」などと尋ねると、直前の文脈を踏まえて回答します。
|
| 740 |
2. **Inventory & Order Query** — 「SKU-1042 は Zone B に何個ある?」「注文 #10007 の状況は?」などを試します。
|
| 741 |
3. **Predictive Maintenance** — プリセットまたはスライダーで1件を判定します。「CSV一括判定」でサンプルCSVをダウンロードし、そのままアップロードして一括判定も試せます。
|
| 742 |
+
4. **Visual Inventory Scan** — 実写真をアップロードするか例示写真を読み込み、「名前で検索」で *pallet, forklift* などを探します(実写真での精度は未測定)。合成ラックのタブでは、ノイズを上げて検出が難しくなる様子を確認できます。
|
| 743 |
5. **Model Evaluation** — すべての図表はテストデータで算出しており、`build_artifacts.py`を再実行すれば同じ数値を再現できます。
|
| 744 |
|
| 745 |
## 評価の信頼性について
|
| 746 |
|
| 747 |
開発中に2件のデータリーク(意図分類での学習/テスト間の文面重複、異常検知でのしきい値への正解率の混入)を発見・修正しました。
|
| 748 |
修正後の指標は、意図分類 正解率 {intent_eval.get('accuracy', 0):.1%}、異常検知 F1 {anomaly_eval.get('f1', 0):.1%} です。
|
| 749 |
+
合成ラックの古典的画像処理はノイズなしでは100%となるため、ノイズを加えた条件(標準ノイズ0.4で完全一致率 {vision_eval.get('exact_match_rate', 0):.0%})を報告しています。実写真に対するYOLOの精度は未測定です。
|
| 750 |
|
| 751 |
## 制約と今後の課題
|
| 752 |
|
| 753 |
- データはすべて**合成**です。実運用ではWMS/WCSのAPIや実際のセンサーログに接続します。
|
| 754 |
+
- 実写真向けのYOLOはゼロショット/汎用モデルであり、倉庫専用データでの微調整・評価は未実施です。精度を保証するには、ラベル付き倉庫画像(約100枚〜)でmAPを測定する必要があります。
|
| 755 |
- Slack通知は`SLACK_WEBHOOK_URL`未設定の場合はプレビューのみで、実際には送信されません。
|
| 756 |
"""
|
| 757 |
|
|
|
|
| 867 |
with gr.Tab("📷 Visual Inventory Scan"):
|
| 868 |
with gr.Accordion("🧪 About this data & function / データ・機能について", open=False):
|
| 869 |
badge_vision = gr.HTML(data_badge(I18N["badge_vision"]["en"]))
|
| 870 |
+
with gr.Tabs():
|
| 871 |
+
with gr.Tab("📸 Real photos (YOLO)"):
|
| 872 |
+
gr.Markdown(
|
| 873 |
+
"Upload a real warehouse photo — or load an openly-licensed example — and run a YOLO "
|
| 874 |
+
"detector. **Standard** mode uses COCO-pretrained YOLOv8 (finds people and vehicles; "
|
| 875 |
+
"COCO has no pallet/forklift/box class). **Find by name** mode uses open-vocabulary "
|
| 876 |
+
"YOLOE: type what to look for (*pallet, cardboard box, forklift…*) and it searches "
|
| 877 |
+
"zero-shot. *The first run downloads model weights (≈1–2 minutes the first time in Find-by-name mode, then about a second).* "
|
| 878 |
+
"Photos are processed in memory and not stored — please don't upload anything sensitive."
|
| 879 |
+
)
|
| 880 |
+
with gr.Row():
|
| 881 |
+
with gr.Column(scale=1):
|
| 882 |
+
real_in = gr.Image(label="Warehouse photo", type="pil", sources=["upload", "clipboard"])
|
| 883 |
+
example_dd = gr.Dropdown(choices=list(EXAMPLE_PHOTOS), value=None,
|
| 884 |
+
label="…or an openly-licensed example (Wikimedia Commons)")
|
| 885 |
+
load_ex_btn = gr.Button("Load example")
|
| 886 |
+
mode_radio = gr.Radio(
|
| 887 |
+
choices=[("Find by name — YOLOE (open-vocabulary)", "open"),
|
| 888 |
+
("Standard objects — YOLOv8 (COCO)", "standard")],
|
| 889 |
+
value="open", label="Detector")
|
| 890 |
+
prompts_in = gr.Textbox(value=DEFAULT_PROMPTS, label="What to look for (comma-separated)", visible=True)
|
| 891 |
+
model_dd = gr.Dropdown(choices=list(MODEL_CHOICES), value=list(MODEL_CHOICES)[0],
|
| 892 |
+
label="YOLOv8 size", visible=False)
|
| 893 |
+
conf_in = gr.Slider(0.05, 0.9, value=0.25, step=0.05, label="Confidence threshold")
|
| 894 |
+
iou_in = gr.Slider(0.3, 0.9, value=0.7, step=0.05,
|
| 895 |
+
label="Overlap suppression (IoU) — lower merges duplicate boxes, but may merge touching pallets")
|
| 896 |
+
detect_btn = gr.Button("🔍 Detect", variant="primary")
|
| 897 |
+
with gr.Column(scale=1):
|
| 898 |
+
real_out = gr.Image(label="Detections", type="pil")
|
| 899 |
+
real_summary = gr.Markdown()
|
| 900 |
+
example_credit = gr.Markdown()
|
| 901 |
+
|
| 902 |
+
mode_radio.change(
|
| 903 |
+
lambda m: (gr.update(visible=(m == "open")), gr.update(visible=(m == "standard"))),
|
| 904 |
+
inputs=mode_radio, outputs=[prompts_in, model_dd])
|
| 905 |
+
load_ex_btn.click(load_example_fn, inputs=example_dd, outputs=[real_in, example_credit])
|
| 906 |
+
detect_btn.click(real_detect_fn, inputs=[real_in, mode_radio, model_dd, prompts_in, conf_in, iou_in],
|
| 907 |
+
outputs=[real_out, real_summary])
|
| 908 |
+
|
| 909 |
+
with gr.Tab("🧪 Synthetic rack (classical-CV baseline)"):
|
| 910 |
+
gr.Markdown(
|
| 911 |
+
"The original baseline: a *non-deep-learning* OpenCV pipeline on synthetic rack images "
|
| 912 |
+
"where ground truth is known, so accuracy can be measured exactly. Raise the noise slider "
|
| 913 |
+
"to watch it degrade the way a dim or grainy real photo would."
|
| 914 |
+
)
|
| 915 |
+
with gr.Row():
|
| 916 |
+
fill_density_in = gr.Slider(0.1, 0.95, value=0.6, step=0.05, label="Rack fill density (scene generation)")
|
| 917 |
+
noise_in = gr.Slider(0.0, 1.0, value=0.3, step=0.05, label="Simulated photographic noise")
|
| 918 |
+
vision_seed_state = gr.State(value=1)
|
| 919 |
+
new_scene_btn = gr.Button("🎲 Generate new rack scene", variant="primary")
|
| 920 |
+
with gr.Row():
|
| 921 |
+
vision_raw_img = gr.Image(label="Rack photo (input)", type="pil")
|
| 922 |
+
vision_annotated_img = gr.Image(label="Detected boxes (output)", type="pil")
|
| 923 |
+
vision_summary = gr.Markdown()
|
| 924 |
+
_vision_io = dict(inputs=[fill_density_in, noise_in, vision_seed_state],
|
| 925 |
+
outputs=[vision_raw_img, vision_annotated_img, vision_summary, vision_seed_state])
|
| 926 |
+
new_scene_btn.click(vision_scan_fn, **_vision_io)
|
| 927 |
+
demo.load(vision_scan_fn, **_vision_io)
|
| 928 |
|
| 929 |
with gr.Tab("📊 Model Evaluation"):
|
| 930 |
with gr.Accordion("🧪 About this data & function / データ・機能について", open=False):
|
|
|
|
| 955 |
gr.Markdown("### 4. Latency Benchmark (CPU, local components)")
|
| 956 |
gr.Image(os.path.join(ASSETS_DIR, "latency_bar.png"), show_label=False, container=False)
|
| 957 |
|
| 958 |
+
gr.Markdown("### 5. Visual Inventory Scan — synthetic-rack classical baseline (measured): exact box-count match **{:.0%}** (noise={}) · count MAE **{:.2f}** boxes".format(
|
| 959 |
vision_eval.get("exact_match_rate", 0), vision_eval.get("standard_noise_level", "?"), vision_eval.get("count_mae", 0)
|
| 960 |
))
|
| 961 |
with gr.Row():
|
|
|
|
| 964 |
with gr.Row():
|
| 965 |
gr.Image(os.path.join(ASSETS_DIR, "vision_sample_raw.png"), label="Sample input", container=True)
|
| 966 |
gr.Image(os.path.join(ASSETS_DIR, "vision_sample_annotated.png"), label="Sample detection", container=True)
|
| 967 |
+
gr.Markdown(
|
| 968 |
+
"### 5b. Real-photo detectors (YOLOv8 / YOLOE) — **not benchmarked**\n"
|
| 969 |
+
"No labelled real warehouse photos were available, so there is deliberately **no accuracy number** here. "
|
| 970 |
+
"COCO benchmark scores published for YOLOv8 describe everyday objects, not pallets or racking, so they "
|
| 971 |
+
"would be misleading to quote. To measure it properly: label ~100 warehouse photos (pallets, forklifts, "
|
| 972 |
+
"people), then report per-class precision / recall / mAP50 — and compare zero-shot YOLOE against a YOLOv8 "
|
| 973 |
+
"fine-tuned on that data."
|
| 974 |
+
)
|
| 975 |
|
| 976 |
with gr.Accordion("📋 Full metrics tables & methodology notes", open=False):
|
| 977 |
gr.Markdown(eval_intent_section_md())
|
requirements.txt
CHANGED
|
@@ -1,4 +1,5 @@
|
|
| 1 |
gradio==5.9.1
|
|
|
|
| 2 |
pydantic==2.10.6
|
| 3 |
scikit-learn==1.7.2
|
| 4 |
pandas==2.2.3
|
|
@@ -6,6 +7,15 @@ numpy==1.26.4
|
|
| 6 |
matplotlib==3.9.2
|
| 7 |
joblib==1.4.2
|
| 8 |
huggingface_hub==0.31.4
|
| 9 |
-
opencv-python-headless==4.10.0.84
|
| 10 |
requests>=2.31.0
|
| 11 |
spaces>=0.30.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
gradio==5.9.1
|
| 2 |
+
# Gradio 5.9.1's API-schema generation crashes on newer pydantic (see DEPLOY.md)
|
| 3 |
pydantic==2.10.6
|
| 4 |
scikit-learn==1.7.2
|
| 5 |
pandas==2.2.3
|
|
|
|
| 7 |
matplotlib==3.9.2
|
| 8 |
joblib==1.4.2
|
| 9 |
huggingface_hub==0.31.4
|
|
|
|
| 10 |
requests>=2.31.0
|
| 11 |
spaces>=0.30.0
|
| 12 |
+
|
| 13 |
+
# --- Visual Inventory Scan: real-photo detection -------------------------------
|
| 14 |
+
# torch is already present on Hugging Face Spaces images, so this stays light there.
|
| 15 |
+
opencv-python==4.10.0.84
|
| 16 |
+
ultralytics==8.4.208
|
| 17 |
+
# YOLOE's text encoder needs Ultralytics' CLIP package (tokenizer); declared here
|
| 18 |
+
# explicitly because Ultralytics' runtime auto-installer is unreliable in containers.
|
| 19 |
+
ftfy
|
| 20 |
+
regex
|
| 21 |
+
git+https://github.com/ultralytics/CLIP.git
|
src/real_vision.py
ADDED
|
@@ -0,0 +1,341 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
real_vision.py
|
| 3 |
+
---------------
|
| 4 |
+
Real-photo detection for the "Visual Inventory Scan" tab.
|
| 5 |
+
|
| 6 |
+
Two detector modes, both from the Ultralytics YOLO family:
|
| 7 |
+
|
| 8 |
+
1. **Standard (COCO-pretrained YOLOv8)** -- fast, but COCO's 80 classes do NOT
|
| 9 |
+
include pallet, forklift, cardboard box or rack. It reliably finds people
|
| 10 |
+
and vehicles (useful for safety checks) but cannot count pallets. The UI
|
| 11 |
+
says so explicitly rather than letting a user assume otherwise.
|
| 12 |
+
|
| 13 |
+
2. **Open-vocabulary (YOLOE)** -- you type what to look for ("pallet,
|
| 14 |
+
cardboard box, forklift") and it searches for those names zero-shot, no
|
| 15 |
+
warehouse-specific training. Accuracy on warehouse imagery is NOT measured
|
| 16 |
+
here (no labelled real warehouse photos were available), so treat counts
|
| 17 |
+
as a starting point, not ground truth.
|
| 18 |
+
|
| 19 |
+
Why YOLOE rather than YOLO-World: both are open-vocabulary YOLO models, but
|
| 20 |
+
YOLO-World's text encoder (CLIP) is hosted on a CDN that could not be
|
| 21 |
+
reached from the development environment, so it could not be verified end
|
| 22 |
+
to end. YOLOE's text encoder (MobileCLIP) ships from the same GitHub
|
| 23 |
+
releases as the weights, and it was tested on a real photo before shipping.
|
| 24 |
+
|
| 25 |
+
Design notes
|
| 26 |
+
------------
|
| 27 |
+
* Weights are downloaded on first use from the official Ultralytics GitHub
|
| 28 |
+
release assets and cached; nothing is bundled in the repo.
|
| 29 |
+
* `ultralytics`/`torch` are imported lazily, so if they're unavailable the
|
| 30 |
+
rest of the app still works and this tab shows a clear message.
|
| 31 |
+
* Example photos are fetched at runtime from Wikimedia Commons (openly
|
| 32 |
+
licensed) and shown with author + license. They are never bundled, so the
|
| 33 |
+
repo contains no third-party images. If the download fails, the user is
|
| 34 |
+
told to upload their own photo.
|
| 35 |
+
* Uploaded images are processed in memory and not stored.
|
| 36 |
+
"""
|
| 37 |
+
|
| 38 |
+
import hashlib
|
| 39 |
+
import os
|
| 40 |
+
import tempfile
|
| 41 |
+
import threading
|
| 42 |
+
import time
|
| 43 |
+
from dataclasses import dataclass, field
|
| 44 |
+
from typing import Dict, List, Optional, Tuple
|
| 45 |
+
|
| 46 |
+
import requests
|
| 47 |
+
from PIL import Image, ImageDraw, ImageFont
|
| 48 |
+
|
| 49 |
+
WEIGHTS_DIR = os.environ.get("YOLO_WEIGHTS_DIR", os.path.join(tempfile.gettempdir(), "yolo_weights"))
|
| 50 |
+
EXAMPLES_DIR = os.path.join(tempfile.gettempdir(), "warehouse_examples")
|
| 51 |
+
ASSETS_BASE = "https://github.com/ultralytics/assets/releases/download/v8.3.0/"
|
| 52 |
+
USER_AGENT = "SmartWarehouseAIDemo/1.0 (portfolio demo; Hugging Face Space)"
|
| 53 |
+
MAX_SIDE = 1600 # downscale very large uploads before inference (CPU friendliness)
|
| 54 |
+
|
| 55 |
+
MODEL_CHOICES = {
|
| 56 |
+
"YOLOv8n (fastest)": "yolov8n.pt",
|
| 57 |
+
"YOLOv8s (more accurate)": "yolov8s.pt",
|
| 58 |
+
}
|
| 59 |
+
OPEN_VOCAB_WEIGHTS = "yoloe-11s-seg.pt"
|
| 60 |
+
TEXT_ENCODER_WEIGHTS = "mobileclip_blt.ts" # ~600 MB, downloaded once on first open-vocabulary use
|
| 61 |
+
DEFAULT_PROMPTS = "pallet, cardboard box, forklift, person, shelving rack"
|
| 62 |
+
|
| 63 |
+
# COCO classes that matter in a warehouse -> a plain-language group
|
| 64 |
+
WAREHOUSE_GROUPS = {
|
| 65 |
+
"person": "people",
|
| 66 |
+
"truck": "vehicles", "car": "vehicles", "bus": "vehicles",
|
| 67 |
+
"motorcycle": "vehicles", "bicycle": "vehicles", "train": "vehicles",
|
| 68 |
+
}
|
| 69 |
+
|
| 70 |
+
# Openly-licensed real photos from Wikimedia Commons (fetched at runtime, never bundled).
|
| 71 |
+
EXAMPLE_PHOTOS: Dict[str, dict] = {
|
| 72 |
+
"Pallet-rack warehouse with forklift": {
|
| 73 |
+
"file": "Modern_warehouse_with_pallet_rack_storage_system.jpg",
|
| 74 |
+
"author": "Axisadman", "license": "CC BY-SA 3.0",
|
| 75 |
+
"license_url": "https://creativecommons.org/licenses/by-sa/3.0/",
|
| 76 |
+
},
|
| 77 |
+
"Warehouse floor: pallets and forklift driver": {
|
| 78 |
+
"file": "Allentown_Project_042.jpg",
|
| 79 |
+
"author": "Chrisking", "license": "CC BY-SA 3.0",
|
| 80 |
+
"license_url": "https://creativecommons.org/licenses/by-sa/3.0/",
|
| 81 |
+
},
|
| 82 |
+
"Stacked CHEP pallets": {
|
| 83 |
+
"file": "576_CHEP_pallets.jpg",
|
| 84 |
+
"author": "train_photos (via Flickr)", "license": "CC BY-SA 2.0",
|
| 85 |
+
"license_url": "https://creativecommons.org/licenses/by-sa/2.0/",
|
| 86 |
+
},
|
| 87 |
+
}
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
@dataclass
|
| 91 |
+
class Detection:
|
| 92 |
+
label: str
|
| 93 |
+
conf: float
|
| 94 |
+
box: Tuple[int, int, int, int] # x1, y1, x2, y2
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
@dataclass
|
| 98 |
+
class RealDetectionResult:
|
| 99 |
+
annotated: Image.Image
|
| 100 |
+
detections: List[Detection]
|
| 101 |
+
counts: Dict[str, int]
|
| 102 |
+
inference_ms: float
|
| 103 |
+
model_name: str
|
| 104 |
+
notes: List[str] = field(default_factory=list)
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
# --------------------------------------------------------------------------
|
| 108 |
+
# Weights + model cache
|
| 109 |
+
# --------------------------------------------------------------------------
|
| 110 |
+
_MODEL_LOCK = threading.Lock()
|
| 111 |
+
_MODELS: Dict[str, object] = {}
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
def _ensure_weights(filename: str) -> str:
|
| 115 |
+
"""Download an Ultralytics release asset once and cache it."""
|
| 116 |
+
os.makedirs(WEIGHTS_DIR, exist_ok=True)
|
| 117 |
+
path = os.path.join(WEIGHTS_DIR, filename)
|
| 118 |
+
if os.path.exists(path) and os.path.getsize(path) > 1_000_000:
|
| 119 |
+
return path
|
| 120 |
+
tmp = path + ".part"
|
| 121 |
+
with requests.get(ASSETS_BASE + filename, stream=True, timeout=60,
|
| 122 |
+
headers={"User-Agent": USER_AGENT}) as r:
|
| 123 |
+
r.raise_for_status()
|
| 124 |
+
with open(tmp, "wb") as f:
|
| 125 |
+
for chunk in r.iter_content(chunk_size=1 << 20):
|
| 126 |
+
f.write(chunk)
|
| 127 |
+
os.replace(tmp, path)
|
| 128 |
+
return path
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
def vision_stack_available() -> Tuple[bool, str]:
|
| 132 |
+
"""Is the optional detector stack (ultralytics + torch) importable?"""
|
| 133 |
+
try:
|
| 134 |
+
import ultralytics # noqa: F401
|
| 135 |
+
return True, ""
|
| 136 |
+
except Exception as e: # noqa: BLE001
|
| 137 |
+
return False, f"{type(e).__name__}: {e}"
|
| 138 |
+
|
| 139 |
+
|
| 140 |
+
def _quiet_settings():
|
| 141 |
+
"""Uploaded photos are user data: turn off Ultralytics' anonymous usage sync."""
|
| 142 |
+
try:
|
| 143 |
+
from ultralytics import settings
|
| 144 |
+
settings.update({"sync": False, "weights_dir": WEIGHTS_DIR})
|
| 145 |
+
except Exception: # noqa: BLE001
|
| 146 |
+
pass
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def _get_yolo(weights_name: str):
|
| 150 |
+
with _MODEL_LOCK:
|
| 151 |
+
if weights_name not in _MODELS:
|
| 152 |
+
from ultralytics import YOLO
|
| 153 |
+
_quiet_settings()
|
| 154 |
+
_MODELS[weights_name] = YOLO(_ensure_weights(weights_name))
|
| 155 |
+
return _MODELS[weights_name]
|
| 156 |
+
|
| 157 |
+
|
| 158 |
+
def _get_open(prompts: Tuple[str, ...]):
|
| 159 |
+
"""YOLOE with a given vocabulary. set_classes() re-encodes the text prompts,
|
| 160 |
+
so cache one model per distinct prompt set (a few at most)."""
|
| 161 |
+
key = "open::" + "|".join(prompts)
|
| 162 |
+
with _MODEL_LOCK:
|
| 163 |
+
if key not in _MODELS:
|
| 164 |
+
from ultralytics import YOLOE
|
| 165 |
+
_quiet_settings()
|
| 166 |
+
# make sure the ~600 MB text encoder is fetched from the official release
|
| 167 |
+
# assets into our cache dir, where ultralytics will look for it
|
| 168 |
+
_ensure_weights(TEXT_ENCODER_WEIGHTS)
|
| 169 |
+
model = YOLOE(_ensure_weights(OPEN_VOCAB_WEIGHTS))
|
| 170 |
+
model.set_classes(list(prompts), model.get_text_pe(list(prompts)))
|
| 171 |
+
_MODELS[key] = model
|
| 172 |
+
# keep the cache from growing without bound if people try many prompt sets
|
| 173 |
+
open_keys = [k for k in _MODELS if k.startswith("open::")]
|
| 174 |
+
for old in open_keys[:-3]:
|
| 175 |
+
_MODELS.pop(old, None)
|
| 176 |
+
return _MODELS[key]
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
# --------------------------------------------------------------------------
|
| 180 |
+
# Drawing
|
| 181 |
+
# --------------------------------------------------------------------------
|
| 182 |
+
_PALETTE = [(231, 76, 60), (52, 152, 219), (46, 204, 113), (241, 196, 15),
|
| 183 |
+
(155, 89, 182), (230, 126, 34), (26, 188, 156), (233, 30, 99)]
|
| 184 |
+
|
| 185 |
+
|
| 186 |
+
def _color_for(label: str) -> Tuple[int, int, int]:
|
| 187 |
+
h = int(hashlib.md5(label.encode()).hexdigest(), 16)
|
| 188 |
+
return _PALETTE[h % len(_PALETTE)]
|
| 189 |
+
|
| 190 |
+
|
| 191 |
+
def _font(size: int):
|
| 192 |
+
try:
|
| 193 |
+
return ImageFont.load_default(size=size) # Pillow >= 10.1
|
| 194 |
+
except TypeError:
|
| 195 |
+
return ImageFont.load_default()
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
def _annotate(image: Image.Image, dets: List[Detection]) -> Image.Image:
|
| 199 |
+
out = image.convert("RGB").copy()
|
| 200 |
+
draw = ImageDraw.Draw(out)
|
| 201 |
+
w, h = out.size
|
| 202 |
+
thick = max(2, round(max(w, h) / 400))
|
| 203 |
+
font = _font(max(12, round(max(w, h) / 55)))
|
| 204 |
+
for d in dets:
|
| 205 |
+
col = _color_for(d.label)
|
| 206 |
+
x1, y1, x2, y2 = d.box
|
| 207 |
+
draw.rectangle([x1, y1, x2, y2], outline=col, width=thick)
|
| 208 |
+
text = f"{d.label} {d.conf:.2f}"
|
| 209 |
+
try:
|
| 210 |
+
tb = draw.textbbox((x1, y1), text, font=font)
|
| 211 |
+
tw, th = tb[2] - tb[0], tb[3] - tb[1]
|
| 212 |
+
except Exception: # noqa: BLE001
|
| 213 |
+
tw, th = len(text) * 7, 12
|
| 214 |
+
ty = max(0, y1 - th - 4)
|
| 215 |
+
draw.rectangle([x1, ty, x1 + tw + 6, ty + th + 4], fill=col)
|
| 216 |
+
draw.text((x1 + 3, ty + 1), text, fill=(255, 255, 255), font=font)
|
| 217 |
+
return out
|
| 218 |
+
|
| 219 |
+
|
| 220 |
+
def _prepare(image: Image.Image) -> Image.Image:
|
| 221 |
+
img = image.convert("RGB")
|
| 222 |
+
if max(img.size) > MAX_SIDE:
|
| 223 |
+
img = img.copy()
|
| 224 |
+
img.thumbnail((MAX_SIDE, MAX_SIDE))
|
| 225 |
+
return img
|
| 226 |
+
|
| 227 |
+
|
| 228 |
+
def parse_prompts(text: str, limit: int = 12) -> Tuple[str, ...]:
|
| 229 |
+
items = []
|
| 230 |
+
for part in (text or "").replace("\n", ",").split(","):
|
| 231 |
+
p = part.strip().lower()
|
| 232 |
+
if p and p not in items:
|
| 233 |
+
items.append(p[:40])
|
| 234 |
+
return tuple(items[:limit])
|
| 235 |
+
|
| 236 |
+
|
| 237 |
+
# --------------------------------------------------------------------------
|
| 238 |
+
# Detection
|
| 239 |
+
# --------------------------------------------------------------------------
|
| 240 |
+
def _run(model, img: Image.Image, conf: float, iou: float = 0.7, names_override=None) -> List[Detection]:
|
| 241 |
+
results = model.predict(img, conf=conf, iou=iou, imgsz=640, verbose=False, device="cpu")
|
| 242 |
+
res = results[0]
|
| 243 |
+
names = names_override or res.names
|
| 244 |
+
dets: List[Detection] = []
|
| 245 |
+
if res.boxes is None:
|
| 246 |
+
return dets
|
| 247 |
+
for box in res.boxes:
|
| 248 |
+
cls_id = int(box.cls[0])
|
| 249 |
+
label = names[cls_id] if isinstance(names, (list, tuple)) else names.get(cls_id, str(cls_id))
|
| 250 |
+
x1, y1, x2, y2 = [int(round(v)) for v in box.xyxy[0].tolist()]
|
| 251 |
+
dets.append(Detection(label=str(label), conf=float(box.conf[0]), box=(x1, y1, x2, y2)))
|
| 252 |
+
dets.sort(key=lambda d: -d.conf)
|
| 253 |
+
return dets
|
| 254 |
+
|
| 255 |
+
|
| 256 |
+
def detect_real(image: Image.Image, mode: str = "standard", model_choice: str = "YOLOv8n (fastest)",
|
| 257 |
+
prompts_text: str = DEFAULT_PROMPTS, conf: float = 0.25, iou: float = 0.7) -> RealDetectionResult:
|
| 258 |
+
"""mode: "standard" (COCO YOLOv8) or "open" (YOLOE, text prompts).
|
| 259 |
+
|
| 260 |
+
`iou` is the overlap threshold for non-max suppression. Lower values merge
|
| 261 |
+
overlapping boxes (fewer duplicate detections of one object) but can merge
|
| 262 |
+
genuinely separate, touching objects such as adjacent pallets -- so it is a
|
| 263 |
+
user-facing control rather than a value tuned on a single test image."""
|
| 264 |
+
img = _prepare(image)
|
| 265 |
+
notes: List[str] = []
|
| 266 |
+
t0 = time.perf_counter()
|
| 267 |
+
if mode == "open":
|
| 268 |
+
prompts = parse_prompts(prompts_text)
|
| 269 |
+
if not prompts:
|
| 270 |
+
raise ValueError("Enter at least one thing to look for, e.g. 'pallet, forklift'.")
|
| 271 |
+
model = _get_open(prompts)
|
| 272 |
+
dets = _run(model, img, conf, iou)
|
| 273 |
+
model_name = f"YOLOE (open-vocabulary) — looking for: {', '.join(prompts)}"
|
| 274 |
+
notes.append("Open-vocabulary detection is zero-shot: no warehouse-specific training, "
|
| 275 |
+
"and its accuracy on warehouse photos has **not** been measured here.")
|
| 276 |
+
else:
|
| 277 |
+
weights = MODEL_CHOICES.get(model_choice, "yolov8n.pt")
|
| 278 |
+
model = _get_yolo(weights)
|
| 279 |
+
dets = _run(model, img, conf, iou)
|
| 280 |
+
model_name = f"{weights} (COCO-pretrained, 80 classes)"
|
| 281 |
+
notes.append("COCO has no *pallet*, *forklift*, *box* or *rack* class, so this mode "
|
| 282 |
+
"can't count them — it finds people and vehicles. Switch to open-vocabulary "
|
| 283 |
+
"mode to search for pallets by name.")
|
| 284 |
+
ms = (time.perf_counter() - t0) * 1000
|
| 285 |
+
counts: Dict[str, int] = {}
|
| 286 |
+
for d in dets:
|
| 287 |
+
counts[d.label] = counts.get(d.label, 0) + 1
|
| 288 |
+
return RealDetectionResult(_annotate(img, dets), dets, counts, ms, model_name, notes)
|
| 289 |
+
|
| 290 |
+
|
| 291 |
+
def summarize(result: RealDetectionResult) -> str:
|
| 292 |
+
if not result.detections:
|
| 293 |
+
body = "**No objects detected** at this confidence threshold — try lowering it."
|
| 294 |
+
else:
|
| 295 |
+
rows = "\n".join(f"| {k} | {v} |" for k, v in sorted(result.counts.items(), key=lambda kv: -kv[1]))
|
| 296 |
+
body = f"**{len(result.detections)} detections**\n\n| Class | Count |\n|---|---|\n{rows}"
|
| 297 |
+
groups: Dict[str, int] = {}
|
| 298 |
+
for lbl, n in result.counts.items():
|
| 299 |
+
g = WAREHOUSE_GROUPS.get(lbl)
|
| 300 |
+
if g:
|
| 301 |
+
groups[g] = groups.get(g, 0) + n
|
| 302 |
+
if groups.get("people"):
|
| 303 |
+
body += (f"\n\n👷 **{groups['people']} person/people in frame** — a production system would "
|
| 304 |
+
f"cross-check this against AGV / forklift lanes for safety alerts.")
|
| 305 |
+
meta = f"\n\n*Model:* {result.model_name} · *Inference:* {result.inference_ms:.0f} ms (measured live on this server)"
|
| 306 |
+
notes = "".join(f"\n\n> ℹ️ {n}" for n in result.notes)
|
| 307 |
+
return body + meta + notes
|
| 308 |
+
|
| 309 |
+
|
| 310 |
+
# --------------------------------------------------------------------------
|
| 311 |
+
# Attributed example photos (Wikimedia Commons, fetched at runtime)
|
| 312 |
+
# --------------------------------------------------------------------------
|
| 313 |
+
def attribution_md(name: str) -> str:
|
| 314 |
+
info = EXAMPLE_PHOTOS[name]
|
| 315 |
+
page = "https://commons.wikimedia.org/wiki/File:" + info["file"]
|
| 316 |
+
return (f"📷 *{name}* — by **{info['author']}**, via [Wikimedia Commons]({page}), "
|
| 317 |
+
f"licensed [{info['license']}]({info['license_url']}). The annotated output is a derivative "
|
| 318 |
+
f"work, shared under the same license.")
|
| 319 |
+
|
| 320 |
+
|
| 321 |
+
def fetch_example(name: str) -> Tuple[Optional[Image.Image], str]:
|
| 322 |
+
"""Returns (image or None, markdown message)."""
|
| 323 |
+
if name not in EXAMPLE_PHOTOS:
|
| 324 |
+
return None, "Pick an example from the list, or upload your own photo."
|
| 325 |
+
info = EXAMPLE_PHOTOS[name]
|
| 326 |
+
os.makedirs(EXAMPLES_DIR, exist_ok=True)
|
| 327 |
+
path = os.path.join(EXAMPLES_DIR, info["file"])
|
| 328 |
+
try:
|
| 329 |
+
if not (os.path.exists(path) and os.path.getsize(path) > 10_000):
|
| 330 |
+
url = f"https://commons.wikimedia.org/wiki/Special:FilePath/{info['file']}?width=1280"
|
| 331 |
+
r = requests.get(url, timeout=30, headers={"User-Agent": USER_AGENT}, allow_redirects=True)
|
| 332 |
+
r.raise_for_status()
|
| 333 |
+
if "image" not in r.headers.get("Content-Type", ""):
|
| 334 |
+
raise ValueError(f"unexpected content type {r.headers.get('Content-Type')}")
|
| 335 |
+
with open(path, "wb") as f:
|
| 336 |
+
f.write(r.content)
|
| 337 |
+
img = Image.open(path).convert("RGB")
|
| 338 |
+
return img, attribution_md(name)
|
| 339 |
+
except Exception as e: # noqa: BLE001
|
| 340 |
+
return None, (f"⚠️ Couldn't download the example photo ({type(e).__name__}). "
|
| 341 |
+
f"This server may be offline — please upload your own warehouse photo instead.")
|