Pro-Coder commited on
Commit
1efaafd
·
verified ·
1 Parent(s): 2d9a408

Upload 44 files

Browse files
Files changed (5) hide show
  1. DEPLOY.md +13 -0
  2. README.md +36 -13
  3. app.py +130 -40
  4. requirements.txt +11 -1
  5. src/real_vision.py +341 -0
DEPLOY.md CHANGED
@@ -128,3 +128,16 @@ branding needs to change.
128
  **Dependency note:** `requirements.txt` pins `pydantic==2.10.6`. Gradio 5.9.1's API-schema generation
129
  crashes on newer pydantic releases (`TypeError: argument of type 'bool' is not iterable`), which would
130
  make the Space fail to start after a rebuild. Don't remove the pin unless you also upgrade Gradio.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
128
  **Dependency note:** `requirements.txt` pins `pydantic==2.10.6`. Gradio 5.9.1's API-schema generation
129
  crashes on newer pydantic releases (`TypeError: argument of type 'bool' is not iterable`), which would
130
  make the Space fail to start after a rebuild. Don't remove the pin unless you also upgrade Gradio.
131
+
132
+ ## Visual Inventory Scan (real photos): deployment notes
133
+
134
+ - `requirements.txt` adds `ultralytics`, `opencv-python`, and Ultralytics' `CLIP` package (from GitHub).
135
+ PyTorch is already present on Hugging Face Spaces images. I could not run a Space build from my
136
+ environment, so **check the build log after pushing**.
137
+ - **If that build fails**, delete the last block of `requirements.txt` (`ultralytics`, `ftfy`, `regex`,
138
+ `git+https://github.com/ultralytics/CLIP.git`) but keep `opencv-python`. I tested this degraded mode:
139
+ the whole app still works, and the real-photo tab explains that the detector isn't available.
140
+ - Memory: open-vocabulary mode keeps a ~600 MB text encoder plus the model in RAM; fine on the free
141
+ 16 GB CPU tier. The first request in that mode takes ~1–2 minutes (weight download).
142
+ - The app calls `device="cpu"` explicitly, so it also runs on ZeroGPU hardware without a GPU.
143
+ - Licence: Ultralytics is AGPL-3.0 — see README.
README.md CHANGED
@@ -7,7 +7,7 @@ sdk: gradio
7
  sdk_version: 5.9.1
8
  app_file: app.py
9
  pinned: false
10
- license: mit
11
  short_description: LLM warehouse copilot, predictive maintenance & CV
12
  ---
13
 
@@ -33,8 +33,13 @@ metrics on held-out test data.
33
  3. **⚠️ Predictive Maintenance** — Isolation Forest anomaly detector (calibrated threshold). Single
34
  readings via sliders, **CSV batch scoring** (sample file provided), and optional **Slack alerts**
35
  (real if `SLACK_WEBHOOK_URL` is set, otherwise a clearly-labelled preview — never claimed as sent).
36
- 4. **📷 Visual Inventory Scan** — a *classical* (non-deep-learning) OpenCV pipeline that counts boxes and
37
- estimates rack occupancy on synthetic rack images, with an adjustable noise slider.
 
 
 
 
 
38
  5. **📊 Model Evaluation** — graphical, held-out metrics for every component, including the vision
39
  pipeline's accuracy-vs-noise curve.
40
  6. **ℹ️ About** — dataset provenance, model function table, architecture, how to test.
@@ -86,6 +91,7 @@ smart-warehouse-ai/
86
  ├── requirements.txt
87
  ├── src/
88
  │ ├── vision_model.py # classical-CV rack scanner (+ synthetic scene generator)
 
89
  │ ├── alerting.py # Slack alert (real webhook or preview) + email preview
90
  │ ├── data_generation.py # synthetic intent / inventory / sensor datasets
91
  │ ├── knowledge_base.py # warehouse-ops knowledge base (RAG source docs)
@@ -114,7 +120,8 @@ benchmark). Headline numbers from the included run:
114
  | Anomaly detector | F1 | ~98% |
115
  | Anomaly detector | ROC-AUC | ~1.00 |
116
  | RAG retriever | Hit-rate@1 / @2 | 88% / 100% |
117
- | Vision (box counting, noise 0.4) | Exact count match | ~88% (100% clean → ~77% at max noise) |
 
118
 
119
  *(Computed on synthetic, held-out test data — see the Evaluation tab for
120
  methodology notes.)*
@@ -146,15 +153,31 @@ number: it's threshold-independent by construction, so it confirms the
146
  model's *ranking* of anomalies was always excellent — the entire story here
147
  is about honestly picking the cutoff, not about detection ability.
148
 
149
- ## License
 
 
 
 
 
 
 
 
 
 
 
 
150
 
151
- MIT — feel free to fork and adapt.
152
 
153
- ## Vision pipeline — honest evaluation notes
 
 
 
 
 
 
154
 
155
- The first detector (Canny edges + dilation) merged every box into one blob; it was replaced with
156
- colour-distance segmentation. On clean renders it then hit a suspicious 100%, so a `noise_level`
157
- (pixel noise, brightness/contrast jitter, per-box colour variation) was added and the headline number is
158
- reported at a moderate noise level alongside a full noise-sweep chart. Occupancy was also redefined from
159
- padded pixel area to *occupied grid cells* to match how ground truth is defined. All scenes are synthetic —
160
- a real deployment would fine-tune a detector (e.g. YOLO) on real photos.
 
7
  sdk_version: 5.9.1
8
  app_file: app.py
9
  pinned: false
10
+ license: agpl-3.0
11
  short_description: LLM warehouse copilot, predictive maintenance & CV
12
  ---
13
 
 
33
  3. **⚠️ Predictive Maintenance** — Isolation Forest anomaly detector (calibrated threshold). Single
34
  readings via sliders, **CSV batch scoring** (sample file provided), and optional **Slack alerts**
35
  (real if `SLACK_WEBHOOK_URL` is set, otherwise a clearly-labelled preview — never claimed as sent).
36
+ 4. **📷 Visual Inventory Scan** — two sub-tabs:
37
+ - **Real photos (YOLO):** upload a warehouse photo (or load an openly-licensed Wikimedia Commons
38
+ example, fetched at runtime with author/licence shown). *Standard* mode = COCO-pretrained YOLOv8
39
+ (finds people/vehicles; COCO has **no** pallet/forklift/box class). *Find by name* mode =
40
+ open-vocabulary **YOLOE** — type "pallet, cardboard box, forklift…" and it searches zero-shot.
41
+ - **Synthetic rack (classical-CV baseline):** the original OpenCV pipeline on synthetic racks with
42
+ known ground truth, so accuracy *can* be measured.
43
  5. **📊 Model Evaluation** — graphical, held-out metrics for every component, including the vision
44
  pipeline's accuracy-vs-noise curve.
45
  6. **ℹ️ About** — dataset provenance, model function table, architecture, how to test.
 
91
  ├── requirements.txt
92
  ├── src/
93
  │ ├── vision_model.py # classical-CV rack scanner (+ synthetic scene generator)
94
+ │ ├── real_vision.py # real-photo detection: YOLOv8 (COCO) + YOLOE (open-vocabulary)
95
  │ ├── alerting.py # Slack alert (real webhook or preview) + email preview
96
  │ ├── data_generation.py # synthetic intent / inventory / sensor datasets
97
  │ ├── knowledge_base.py # warehouse-ops knowledge base (RAG source docs)
 
120
  | Anomaly detector | F1 | ~98% |
121
  | Anomaly detector | ROC-AUC | ~1.00 |
122
  | RAG retriever | Hit-rate@1 / @2 | 88% / 100% |
123
+ | Synthetic-rack CV baseline (noise 0.4) | Exact box-count match | ~88% (100% clean → ~77% at max noise) |
124
+ | Real-photo YOLO / YOLOE | — | **Not benchmarked** (no labelled warehouse photos available) |
125
 
126
  *(Computed on synthetic, held-out test data — see the Evaluation tab for
127
  methodology notes.)*
 
153
  model's *ranking* of anomalies was always excellent — the entire story here
154
  is about honestly picking the cutoff, not about detection ability.
155
 
156
+ ## Real-photo detection: what was and wasn't verified
157
+
158
+ | | Status |
159
+ |---|---|
160
+ | YOLOv8n / YOLOv8s (COCO) on a real photo | ✅ run end-to-end; detections visually checked |
161
+ | YOLOE open-vocabulary on a real photo | ✅ run end-to-end through the app's API; one duplicate-box over-count observed, hence the user-facing IoU slider |
162
+ | Accuracy on **warehouse** photos | ❌ **Not measured** — no labelled warehouse images were available |
163
+ | Wikimedia example photos | ⚠️ Fetched at runtime on the Space; my development sandbox couldn't reach Wikimedia, so these were *not* run through the detector before delivery (the failure path *was* tested) |
164
+ | YOLO-World / OWLv2 / Grounding DINO | Not shipped — their weights are hosted where I couldn't verify them |
165
+
166
+ Photos are processed in memory and never stored; Ultralytics' anonymous usage sync is switched off.
167
+ First use downloads weights from the official Ultralytics GitHub releases (≈630 MB for open-vocabulary
168
+ mode, once per container).
169
 
170
+ ## Image attributions
171
 
172
+ Example photos are **not bundled**; they're fetched from Wikimedia Commons when selected, and shown with
173
+ author and licence:
174
+ - *Modern warehouse with pallet rack storage system.jpg* — Axisadman, CC BY-SA 3.0
175
+ - *Allentown Project 042.jpg* — Chrisking, CC BY-SA 3.0
176
+ - *576 CHEP pallets.jpg* — train_photos (Flickr), CC BY-SA 2.0
177
+
178
+ ## License
179
 
180
+ **AGPL-3.0.** The real-photo detector uses Ultralytics YOLO, whose code and pretrained weights are
181
+ AGPL-3.0; serving them as a network app means the combined project is distributed under the same terms
182
+ (source is available in this repository). If you want to reuse the project in a closed-source product,
183
+ Ultralytics sells a commercial licence, or you can swap in a differently-licensed detector.
 
 
app.py CHANGED
@@ -29,6 +29,10 @@ from src.anomaly_model import load_artifacts as load_anomaly_artifacts
29
  from src.anomaly_model import score_reading
30
  from src.alerting import send_anomaly_alert
31
  from src.vision_model import generate_rack_scene, detect_boxes
 
 
 
 
32
  from src.data_generation import generate_inventory_db, generate_orders_db
33
  from src.intent_model import INTENT_DESCRIPTIONS, load_pipeline, predict as intent_predict
34
  from src.inventory_db import query_inventory, query_orders
@@ -319,6 +323,30 @@ def vision_scan_fn(fill_density, noise_level, seed_state):
319
  return scene.image, result.annotated_image, summary, seed + 1
320
 
321
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
322
  # ==========================================================================
323
  # TAB 4 -- Model Evaluation
324
  # ==========================================================================
@@ -512,7 +540,19 @@ model by default.
512
  ## Extended capabilities
513
 
514
  - **Multi-turn conversation** — the assistant keeps the last few turns, so follow-ups like *"what about Zone D instead?"* work.
515
- - **📷 Visual Inventory Scan** — a classical (non-deep-learning) computer-vision pipeline counts boxes and estimates occupancy on synthetic rack images. Evaluated honestly: **100%** on clean renders, but it degrades as simulated photographic noise rises, so the headline number is reported at a moderate noise level (see Model Evaluation).
 
 
 
 
 
 
 
 
 
 
 
 
516
  - **Batch CSV scoring** — upload a CSV of sensor readings (a sample is provided) to score a whole shift's logs at once, and download the results.
517
  - **Slack alerting** — anomalies can trigger a Slack alert. Real if a `SLACK_WEBHOOK_URL` secret is set; otherwise a clearly-labelled preview that is *never* presented as sent.
518
  - **Bilingual UI** — switch the explanatory text between English and 日本語 with the selector at the top. (Charts and metric tables stay in English; the assistant answers in whatever language you ask in.)
@@ -622,21 +662,23 @@ I18N = {
622
  "異常時はSlack通知が可能です(<code>SLACK_WEBHOOK_URL</code>設定時は実送信、未設定時はプレビュー表示)。",
623
  },
624
  "badge_vision": {
625
- "en": "🧪 <b>Source:</b> synthetic rack-scene images generated by the developer's code (<code>src/vision_model.py</code>) — not real "
626
- "warehouse photos (none were available to license).<br>"
627
- "<b>What it describes:</b> a simplified front view of a 3-row × 6-column storage rack with randomly placed box-like shapes "
628
- "standing in for pallets/cartons, plus adjustable simulated photographic noise.<br>"
629
- "<b>Function:</b> a <i>classical</i> (non-deep-learning) computer-vision pipeline — colour-based background/shelf segmentation, "
630
- "morphological cleanup, connected components, geometric filtering — counts boxes and estimates rack occupancy. Deliberately the "
631
- "opposite of a pretrained detector: zero model downloads, fully explainable, instant on CPU. The Model Evaluation tab shows how "
632
- "accuracy degrades as noise rises.",
633
- "ja": "🧪 <b>データの出典:</b>開発者のコード(<code>src/vision_model.py</code>)で生成した合成のラック画像。実際の倉庫写真ではありません"
634
- "(利用可能なライセンス素材がないため)。<br>"
635
- "<b>内容:</b>3段×6列の保管ラックを正面から見た簡易画像で、パレット/ケースに見立てた箱がランダムに配置されます。"
636
- "撮影ノイズの強さも調整できます。<br>"
637
- "<b>機能:</b>深層学習を使わない<i>古典的</i>な画像処理パイプライン(色によるラック背景の分離 → モルフォロジー処理 → 連結成分抽出 → "
638
- "形状フィルタ)で、箱の数と棚の占有率を推定します。学習済みモデルのダウンロードが不要で、処理過程を説明でき、CPUで即時に動作します。"
639
- "ノイズ増加時の精度低下は「Model Evaluation」タブで確認できます。",
 
 
640
  },
641
  "badge_eval": {
642
  "en": "🧪 All charts below are computed on <b>held-out test data</b> by <code>build_artifacts.py</code> — fully reproducible, not cherry-picked.",
@@ -663,7 +705,7 @@ AS/RS、コンベヤ・仕分け、AGV/AMR、WMS/WCSといった倉庫自動化
663
  | **意図分類**による問い合わせの振り分け(8カテゴリ) | AI Assistant / Inventory | TF-IDF + ロジスティック回帰 |
664
  | 在庫・注文の**自然文照会** | Inventory & Order Query | 正規表現によるスロット抽出 + 意図分類 |
665
  | センサーデータの**予知保全**(1件判定/CSV一括判定/Slack通知) | Predictive Maintenance | Isolation Forest(教師なし)+ 較正済みしきい値 |
666
- | ラック画像の**箱数カウントと占有率推定** | Visual Inventory Scan | 古典的画像処理(OpenCV) |
667
  | 全コンポーネントの**再現可能な評価** | Model Evaluation | scikit-learn指標、ホールドアウト検証 |
668
 
669
  ## データセットの概要
@@ -690,26 +732,26 @@ AS/RS、コンベヤ・仕分け、AGV/AMR、WMS/WCSといった倉庫自動化
690
  | LLMアシスタント | 根拠に基づく自然な回答を生成 | ホスト型LLM(複数モデルのフォールバック) | 質問 + 根拠 + 会話履歴 → 回答 |
691
  | 在庫・注文照会 | 質問をテーブル検索に変換 | 正規表現 + 意図分類 | 文章 → 絞り込み結果 |
692
  | 異常検知 | 設備の異常な計測値を早期に検出 | Isolation Forest + 較正済みしきい値 | 4つのセンサー値 → 正常/異常 + スコア |
693
- | ラックスキャナ | 箱の数と占有率を推定 | 色分離 + 輪郭抽出(古典的画像処理) | 画像 → 箱数 + 占有率 + 検出枠 |
694
 
695
  ## 試し方
696
 
697
  1. **AI Assistant** — 例の質問を選ぶか自由に入力します。続けて「Zone Dの場合は?」などと尋ねると、直前の文脈を踏まえて回答します。
698
  2. **Inventory & Order Query** — 「SKU-1042 は Zone B に何個ある?」「注文 #10007 の状況は?」などを試します。
699
  3. **Predictive Maintenance** — プリセットまたはスライダーで1件を判定します。「CSV一括判定」でサンプルCSVをダウンロードし、そのままアップロードして一括判定も試せます。
700
- 4. **Visual Inventory Scan** — 「新しいラック画像を生成」を押し、ノイズを上げて検出が難しくなる様子を確認します。
701
  5. **Model Evaluation** — すべての図表はテストデータで算出しており、`build_artifacts.py`を再実行すれば同じ数値を再現できます。
702
 
703
  ## 評価の信頼性について
704
 
705
  開発中に2件のデータリーク(意図分類での学習/テスト間の文面重複、異常検知でのしきい値への正解率の混入)を発見・修正しました。
706
  修正後の指標は、意図分類 正解率 {intent_eval.get('accuracy', 0):.1%}、異常検知 F1 {anomaly_eval.get('f1', 0):.1%} です。
707
- 画像パイプラインも、ノイズなしでは100%となるため、ノイズを加えた条件(標準ノイズ0.4で完全一致率 {vision_eval.get('exact_match_rate', 0):.0%})を報告しています。
708
 
709
  ## 制約と今後の課題
710
 
711
  - データはすべて**合成**です。実運用ではWMS/WCSのAPIや実際のセンサーログに接続します。
712
- - 画像は単純化した合成ラックであり、実写真への適用には学習済み検出器(YOLO等)の微調整が現実的です。
713
  - Slack通知は`SLACK_WEBHOOK_URL`未設定の場合はプレビューのみで、実際には送信されません。
714
  """
715
 
@@ -825,24 +867,64 @@ with gr.Blocks(theme=gr.themes.Soft(primary_hue="blue"), css=CUSTOM_CSS, title="
825
  with gr.Tab("📷 Visual Inventory Scan"):
826
  with gr.Accordion("🧪 About this data & function / データ・機能について", open=False):
827
  badge_vision = gr.HTML(data_badge(I18N["badge_vision"]["en"]))
828
- gr.Markdown(
829
- "Generate a synthetic rack photo and watch the pipeline count boxes and estimate "
830
- "occupancy. Raise the noise slider to see detection get harder, the way a dim or "
831
- "grainy real photo would."
832
- )
833
- with gr.Row():
834
- fill_density_in = gr.Slider(0.1, 0.95, value=0.6, step=0.05, label="Rack fill density (scene generation)")
835
- noise_in = gr.Slider(0.0, 1.0, value=0.3, step=0.05, label="Simulated photographic noise")
836
- vision_seed_state = gr.State(value=1)
837
- new_scene_btn = gr.Button("🎲 Generate new rack scene", variant="primary")
838
- with gr.Row():
839
- vision_raw_img = gr.Image(label="Rack photo (input)", type="pil")
840
- vision_annotated_img = gr.Image(label="Detected boxes (output)", type="pil")
841
- vision_summary = gr.Markdown()
842
- _vision_io = dict(inputs=[fill_density_in, noise_in, vision_seed_state],
843
- outputs=[vision_raw_img, vision_annotated_img, vision_summary, vision_seed_state])
844
- new_scene_btn.click(vision_scan_fn, **_vision_io)
845
- demo.load(vision_scan_fn, **_vision_io)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
846
 
847
  with gr.Tab("📊 Model Evaluation"):
848
  with gr.Accordion("🧪 About this data & function / データ・機能について", open=False):
@@ -873,7 +955,7 @@ with gr.Blocks(theme=gr.themes.Soft(primary_hue="blue"), css=CUSTOM_CSS, title="
873
  gr.Markdown("### 4. Latency Benchmark (CPU, local components)")
874
  gr.Image(os.path.join(ASSETS_DIR, "latency_bar.png"), show_label=False, container=False)
875
 
876
- gr.Markdown("### 5. Visual Inventory Scan — exact box-count match: **{:.0%}** (noise={}) · count MAE: **{:.2f}** boxes".format(
877
  vision_eval.get("exact_match_rate", 0), vision_eval.get("standard_noise_level", "?"), vision_eval.get("count_mae", 0)
878
  ))
879
  with gr.Row():
@@ -882,6 +964,14 @@ with gr.Blocks(theme=gr.themes.Soft(primary_hue="blue"), css=CUSTOM_CSS, title="
882
  with gr.Row():
883
  gr.Image(os.path.join(ASSETS_DIR, "vision_sample_raw.png"), label="Sample input", container=True)
884
  gr.Image(os.path.join(ASSETS_DIR, "vision_sample_annotated.png"), label="Sample detection", container=True)
 
 
 
 
 
 
 
 
885
 
886
  with gr.Accordion("📋 Full metrics tables & methodology notes", open=False):
887
  gr.Markdown(eval_intent_section_md())
 
29
  from src.anomaly_model import score_reading
30
  from src.alerting import send_anomaly_alert
31
  from src.vision_model import generate_rack_scene, detect_boxes
32
+ from src.real_vision import (
33
+ DEFAULT_PROMPTS, EXAMPLE_PHOTOS, MODEL_CHOICES, detect_real, fetch_example,
34
+ summarize as summarize_real, vision_stack_available,
35
+ )
36
  from src.data_generation import generate_inventory_db, generate_orders_db
37
  from src.intent_model import INTENT_DESCRIPTIONS, load_pipeline, predict as intent_predict
38
  from src.inventory_db import query_inventory, query_orders
 
323
  return scene.image, result.annotated_image, summary, seed + 1
324
 
325
 
326
+ def real_detect_fn(image, mode, model_choice, prompts_text, conf, iou=0.7):
327
+ """Run a real-photo detector (YOLO) on an uploaded / example image."""
328
+ if image is None:
329
+ return None, "⬆️ Upload a photo, or load an example, first."
330
+ ok, why = vision_stack_available()
331
+ if not ok:
332
+ return None, (f"⚠️ The detector stack (ultralytics + PyTorch) isn't available on this server "
333
+ f"(`{why}`). The rest of the app is unaffected.")
334
+ try:
335
+ res = detect_real(image, mode=mode, model_choice=model_choice, prompts_text=prompts_text, conf=conf, iou=iou)
336
+ except ValueError as e: # bad user input (e.g. empty prompt box) -> plain guidance
337
+ return None, f"✏️ {e}"
338
+ except Exception as e: # noqa: BLE001
339
+ return None, (f"⚠️ Detection failed: `{type(e).__name__}: {e}`\n\n"
340
+ f"The first run downloads model weights from GitHub, so a network problem there "
341
+ f"is the most common cause — try again, or switch detector mode.")
342
+ return res.annotated, summarize_real(res)
343
+
344
+
345
+ def load_example_fn(name):
346
+ img, msg = fetch_example(name)
347
+ return img, msg
348
+
349
+
350
  # ==========================================================================
351
  # TAB 4 -- Model Evaluation
352
  # ==========================================================================
 
540
  ## Extended capabilities
541
 
542
  - **Multi-turn conversation** — the assistant keeps the last few turns, so follow-ups like *"what about Zone D instead?"* work.
543
+ - **📷 Visual Inventory Scan** — runs YOLO on **real warehouse photos** (upload your own, or load an openly-licensed example). *Standard* mode (COCO-pretrained YOLOv8) finds people and vehicles but has no pallet/forklift/box class; *Find by name* mode (open-vocabulary YOLOE) searches for whatever you type, zero-shot. **Honest limit:** accuracy on real warehouse photos is *not measured* (no labelled warehouse images were available). A classical-CV baseline on synthetic racks, where ground truth is known, *is* measured: 100% on clean renders, degrading under simulated noise (see Model Evaluation).
544
+
545
+ **Detector options considered for real photos** (latency = rough figures from a development CPU, after model load):
546
+
547
+ | Option | Finds pallets / forklifts / boxes? | Weight & speed | Status |
548
+ |---|---|---|---|
549
+ | YOLOv8 (COCO) | **No** — no such classes; finds people & vehicles | 6–22 MB, ~75 ms | ✅ Shipped; run on a real photo |
550
+ | YOLOE (open-vocabulary) | Yes, zero-shot via text prompts | 28 MB + ~600 MB text encoder, ~250 ms; first load ≈ 1–2 min | ✅ Shipped; run on a real photo (not on warehouse photos) |
551
+ | YOLO-World (open-vocabulary) | Yes, zero-shot | text encoder hosted on a CDN I couldn't reach | ❌ Not shipped — couldn't verify it |
552
+ | OWLv2 / Grounding DINO | Yes, zero-shot, often stronger | hundreds of MB, seconds per image on CPU; weights on the Hugging Face Hub, unreachable from my environment | ❌ Not shipped — couldn't verify it |
553
+ | YOLO fine-tuned on a warehouse dataset | Best accuracy | needs licensed, labelled warehouse photos + training | ⏳ Recommended next step |
554
+ | Classical CV (OpenCV) | Only clean synthetic racks | tiny | ✅ Kept as a *measured* baseline |
555
+
556
  - **Batch CSV scoring** — upload a CSV of sensor readings (a sample is provided) to score a whole shift's logs at once, and download the results.
557
  - **Slack alerting** — anomalies can trigger a Slack alert. Real if a `SLACK_WEBHOOK_URL` secret is set; otherwise a clearly-labelled preview that is *never* presented as sent.
558
  - **Bilingual UI** — switch the explanatory text between English and 日本語 with the selector at the top. (Charts and metric tables stay in English; the assistant answers in whatever language you ask in.)
 
662
  "異常時はSlack通知が可能です(<code>SLACK_WEBHOOK_URL</code>設定時は実送信、未設定時はプレビュー表示)。",
663
  },
664
  "badge_vision": {
665
+ "en": "🧪 <b>Source:</b> <b>Real photos:</b> anything you upload (processed in memory, not stored), plus three openly-licensed "
666
+ "example photos fetched at runtime from Wikimedia Commons with author and license shown — no third-party images are bundled "
667
+ "in this repo. <b>Synthetic baseline:</b> rack images generated by <code>src/vision_model.py</code>, where ground truth is known.<br>"
668
+ "<b>What it describes:</b> real warehouse scenes (racking, pallets, forklifts, people) for the YOLO detectors; a simplified 3×6 storage rack "
669
+ "with box shapes for the baseline.<br>"
670
+ "<b>Function:</b> <i>Standard</i> mode runs COCO-pretrained YOLOv8, which finds people and vehicles but has <b>no pallet, forklift or box "
671
+ "class</b>. <i>Find by name</i> mode runs open-vocabulary YOLOE: type what to look for and it searches zero-shot. "
672
+ "<b>Accuracy on real warehouse photos has not been measured</b> — no labelled warehouse images were available — so treat counts as a "
673
+ "starting point. The classical-CV baseline on synthetic racks <i>is</i> measured (see Model Evaluation).",
674
+ "ja": "🧪 <b>データの出典:</b><b>実写真:</b>アップロードした画像(メモリ上で処理し、保存しません)と、Wikimedia Commonsから実行時に取得する"
675
+ "オープンライセンスの例示写真3枚(作者・ライセンスを表示)。第三者の画像はリポジトリに同梱していません。"
676
+ "<b>合成ベースライン:</b><code>src/vision_model.py</code>で生成した、正解が既知のラック画像。<br>"
677
+ "<b>内容:</b>YOLO検出器向けの実際の倉庫シーン(ラック、パレット、フォークリフト、作業者)と、ベースライン用の簡易な3×6ラック画像。<br>"
678
+ "<b>機能:</b><i>標準</i>モードはCOCO学習済みYOLOv8を使用し、人と車両は検出できますが、<b>パレット・フォークリフト・箱のクラスはありません</b>。"
679
+ "<i>名前で検索</i>モードはオープン語彙のYOLOEで、探したい物を文字で入力してゼロショット検出します。"
680
+ "<b>実際の倉庫写真での精度は未測定</b>です(ラベル付き倉庫画像が入手できなかったため)。件数は目安としてご利用ください。"
681
+ "合成ラックでの古典的画像処理ベースラインは測定済みです(Model Evaluationタブ参照)。",
682
  },
683
  "badge_eval": {
684
  "en": "🧪 All charts below are computed on <b>held-out test data</b> by <code>build_artifacts.py</code> — fully reproducible, not cherry-picked.",
 
705
  | **意図分類**による問い合わせの振り分け(8カテゴリ) | AI Assistant / Inventory | TF-IDF + ロジスティック回帰 |
706
  | 在庫・注文の**自然文照会** | Inventory & Order Query | 正規表現によるスロット抽出 + 意図分類 |
707
  | センサーデータの**予知保全**(1件判定/CSV一括判定/Slack通知) | Predictive Maintenance | Isolation Forest(教師なし)+ 較正済みしきい値 |
708
+ | **実写真の物体検出**(人・車両/名前指定でパレット等)+合成ラックのベースライン | Visual Inventory Scan | YOLOv8 / YOLOE + 古典的画像処理 |
709
  | 全コンポーネントの**再現可能な評価** | Model Evaluation | scikit-learn指標、ホールドアウト検証 |
710
 
711
  ## データセットの概要
 
732
  | LLMアシスタント | 根拠に基づく自然な回答を生成 | ホスト型LLM(複数モデルのフォールバック) | 質問 + 根拠 + 会話履歴 → 回答 |
733
  | 在庫・注文照会 | 質問をテーブル検索に変換 | 正規表現 + 意図分類 | 文章 → 絞り込み結果 |
734
  | 異常検知 | 設備の異常な計測値を早期に検出 | Isolation Forest + 較正済みしきい値 | 4つのセンサー値 → 正常/異常 + スコア |
735
+ | 画像検出器 | 実写真から物体を検出 | YOLOv8(COCO)/YOLOE(名前指定)、ベースラインは古典的画像処理 | 画像 → 検出枠 + クラス別件数 |
736
 
737
  ## 試し方
738
 
739
  1. **AI Assistant** — 例の質問を選ぶか自由に入力します。続けて「Zone Dの場合は?」などと尋ねると、直前の文脈を踏まえて回答します。
740
  2. **Inventory & Order Query** — 「SKU-1042 は Zone B に何個ある?」「注文 #10007 の状況は?」などを試します。
741
  3. **Predictive Maintenance** — プリセットまたはスライダーで1件を判定します。「CSV一括判定」でサンプルCSVをダウンロードし、そのままアップロードして一括判定も試せます。
742
+ 4. **Visual Inventory Scan** — 実写真をアップロードするか例示写真を読み込み、「名前で検索」で *pallet, forklift* などを探します(実写真での精度は未測定)。合成ラックのタブでは、ノイズを上げて検出が難しくなる様子を確認できます。
743
  5. **Model Evaluation** — すべての図表はテストデータで算出しており、`build_artifacts.py`を再実行すれば同じ数値を再現できます。
744
 
745
  ## 評価の信頼性について
746
 
747
  開発中に2件のデータリーク(意図分類での学習/テスト間の文面重複、異常検知でのしきい値への正解率の混入)を発見・修正しました。
748
  修正後の指標は、意図分類 正解率 {intent_eval.get('accuracy', 0):.1%}、異常検知 F1 {anomaly_eval.get('f1', 0):.1%} です。
749
+ 合成ラックの古典的画像処理はノイズなしでは100%となるため、ノイズを加えた条件(標準ノイズ0.4で完全一致率 {vision_eval.get('exact_match_rate', 0):.0%})を報告しています。実写真に対するYOLOの精度は未測定です。
750
 
751
  ## 制約と今後の課題
752
 
753
  - データはすべて**合成**です。実運用ではWMS/WCSのAPIや実際のセンサーログに接続します。
754
+ - 実写真向けのYOLOはゼロショット/汎用モデルであり、倉庫専用データでの微調整・評価は未実施です。精度を保証するには、ラベル付き倉庫画像(約100枚〜)でmAPを測定する必要があります。
755
  - Slack通知は`SLACK_WEBHOOK_URL`未設定の場合はプレビューのみで、実際には送信されません。
756
  """
757
 
 
867
  with gr.Tab("📷 Visual Inventory Scan"):
868
  with gr.Accordion("🧪 About this data & function / データ・機能について", open=False):
869
  badge_vision = gr.HTML(data_badge(I18N["badge_vision"]["en"]))
870
+ with gr.Tabs():
871
+ with gr.Tab("📸 Real photos (YOLO)"):
872
+ gr.Markdown(
873
+ "Upload a real warehouse photo — or load an openly-licensed example — and run a YOLO "
874
+ "detector. **Standard** mode uses COCO-pretrained YOLOv8 (finds people and vehicles; "
875
+ "COCO has no pallet/forklift/box class). **Find by name** mode uses open-vocabulary "
876
+ "YOLOE: type what to look for (*pallet, cardboard box, forklift…*) and it searches "
877
+ "zero-shot. *The first run downloads model weights (≈1–2 minutes the first time in Find-by-name mode, then about a second).* "
878
+ "Photos are processed in memory and not stored — please don't upload anything sensitive."
879
+ )
880
+ with gr.Row():
881
+ with gr.Column(scale=1):
882
+ real_in = gr.Image(label="Warehouse photo", type="pil", sources=["upload", "clipboard"])
883
+ example_dd = gr.Dropdown(choices=list(EXAMPLE_PHOTOS), value=None,
884
+ label="…or an openly-licensed example (Wikimedia Commons)")
885
+ load_ex_btn = gr.Button("Load example")
886
+ mode_radio = gr.Radio(
887
+ choices=[("Find by name — YOLOE (open-vocabulary)", "open"),
888
+ ("Standard objects — YOLOv8 (COCO)", "standard")],
889
+ value="open", label="Detector")
890
+ prompts_in = gr.Textbox(value=DEFAULT_PROMPTS, label="What to look for (comma-separated)", visible=True)
891
+ model_dd = gr.Dropdown(choices=list(MODEL_CHOICES), value=list(MODEL_CHOICES)[0],
892
+ label="YOLOv8 size", visible=False)
893
+ conf_in = gr.Slider(0.05, 0.9, value=0.25, step=0.05, label="Confidence threshold")
894
+ iou_in = gr.Slider(0.3, 0.9, value=0.7, step=0.05,
895
+ label="Overlap suppression (IoU) — lower merges duplicate boxes, but may merge touching pallets")
896
+ detect_btn = gr.Button("🔍 Detect", variant="primary")
897
+ with gr.Column(scale=1):
898
+ real_out = gr.Image(label="Detections", type="pil")
899
+ real_summary = gr.Markdown()
900
+ example_credit = gr.Markdown()
901
+
902
+ mode_radio.change(
903
+ lambda m: (gr.update(visible=(m == "open")), gr.update(visible=(m == "standard"))),
904
+ inputs=mode_radio, outputs=[prompts_in, model_dd])
905
+ load_ex_btn.click(load_example_fn, inputs=example_dd, outputs=[real_in, example_credit])
906
+ detect_btn.click(real_detect_fn, inputs=[real_in, mode_radio, model_dd, prompts_in, conf_in, iou_in],
907
+ outputs=[real_out, real_summary])
908
+
909
+ with gr.Tab("🧪 Synthetic rack (classical-CV baseline)"):
910
+ gr.Markdown(
911
+ "The original baseline: a *non-deep-learning* OpenCV pipeline on synthetic rack images "
912
+ "where ground truth is known, so accuracy can be measured exactly. Raise the noise slider "
913
+ "to watch it degrade the way a dim or grainy real photo would."
914
+ )
915
+ with gr.Row():
916
+ fill_density_in = gr.Slider(0.1, 0.95, value=0.6, step=0.05, label="Rack fill density (scene generation)")
917
+ noise_in = gr.Slider(0.0, 1.0, value=0.3, step=0.05, label="Simulated photographic noise")
918
+ vision_seed_state = gr.State(value=1)
919
+ new_scene_btn = gr.Button("🎲 Generate new rack scene", variant="primary")
920
+ with gr.Row():
921
+ vision_raw_img = gr.Image(label="Rack photo (input)", type="pil")
922
+ vision_annotated_img = gr.Image(label="Detected boxes (output)", type="pil")
923
+ vision_summary = gr.Markdown()
924
+ _vision_io = dict(inputs=[fill_density_in, noise_in, vision_seed_state],
925
+ outputs=[vision_raw_img, vision_annotated_img, vision_summary, vision_seed_state])
926
+ new_scene_btn.click(vision_scan_fn, **_vision_io)
927
+ demo.load(vision_scan_fn, **_vision_io)
928
 
929
  with gr.Tab("📊 Model Evaluation"):
930
  with gr.Accordion("🧪 About this data & function / データ・機能について", open=False):
 
955
  gr.Markdown("### 4. Latency Benchmark (CPU, local components)")
956
  gr.Image(os.path.join(ASSETS_DIR, "latency_bar.png"), show_label=False, container=False)
957
 
958
+ gr.Markdown("### 5. Visual Inventory Scan — synthetic-rack classical baseline (measured): exact box-count match **{:.0%}** (noise={}) · count MAE **{:.2f}** boxes".format(
959
  vision_eval.get("exact_match_rate", 0), vision_eval.get("standard_noise_level", "?"), vision_eval.get("count_mae", 0)
960
  ))
961
  with gr.Row():
 
964
  with gr.Row():
965
  gr.Image(os.path.join(ASSETS_DIR, "vision_sample_raw.png"), label="Sample input", container=True)
966
  gr.Image(os.path.join(ASSETS_DIR, "vision_sample_annotated.png"), label="Sample detection", container=True)
967
+ gr.Markdown(
968
+ "### 5b. Real-photo detectors (YOLOv8 / YOLOE) — **not benchmarked**\n"
969
+ "No labelled real warehouse photos were available, so there is deliberately **no accuracy number** here. "
970
+ "COCO benchmark scores published for YOLOv8 describe everyday objects, not pallets or racking, so they "
971
+ "would be misleading to quote. To measure it properly: label ~100 warehouse photos (pallets, forklifts, "
972
+ "people), then report per-class precision / recall / mAP50 — and compare zero-shot YOLOE against a YOLOv8 "
973
+ "fine-tuned on that data."
974
+ )
975
 
976
  with gr.Accordion("📋 Full metrics tables & methodology notes", open=False):
977
  gr.Markdown(eval_intent_section_md())
requirements.txt CHANGED
@@ -1,4 +1,5 @@
1
  gradio==5.9.1
 
2
  pydantic==2.10.6
3
  scikit-learn==1.7.2
4
  pandas==2.2.3
@@ -6,6 +7,15 @@ numpy==1.26.4
6
  matplotlib==3.9.2
7
  joblib==1.4.2
8
  huggingface_hub==0.31.4
9
- opencv-python-headless==4.10.0.84
10
  requests>=2.31.0
11
  spaces>=0.30.0
 
 
 
 
 
 
 
 
 
 
 
1
  gradio==5.9.1
2
+ # Gradio 5.9.1's API-schema generation crashes on newer pydantic (see DEPLOY.md)
3
  pydantic==2.10.6
4
  scikit-learn==1.7.2
5
  pandas==2.2.3
 
7
  matplotlib==3.9.2
8
  joblib==1.4.2
9
  huggingface_hub==0.31.4
 
10
  requests>=2.31.0
11
  spaces>=0.30.0
12
+
13
+ # --- Visual Inventory Scan: real-photo detection -------------------------------
14
+ # torch is already present on Hugging Face Spaces images, so this stays light there.
15
+ opencv-python==4.10.0.84
16
+ ultralytics==8.4.208
17
+ # YOLOE's text encoder needs Ultralytics' CLIP package (tokenizer); declared here
18
+ # explicitly because Ultralytics' runtime auto-installer is unreliable in containers.
19
+ ftfy
20
+ regex
21
+ git+https://github.com/ultralytics/CLIP.git
src/real_vision.py ADDED
@@ -0,0 +1,341 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ real_vision.py
3
+ ---------------
4
+ Real-photo detection for the "Visual Inventory Scan" tab.
5
+
6
+ Two detector modes, both from the Ultralytics YOLO family:
7
+
8
+ 1. **Standard (COCO-pretrained YOLOv8)** -- fast, but COCO's 80 classes do NOT
9
+ include pallet, forklift, cardboard box or rack. It reliably finds people
10
+ and vehicles (useful for safety checks) but cannot count pallets. The UI
11
+ says so explicitly rather than letting a user assume otherwise.
12
+
13
+ 2. **Open-vocabulary (YOLOE)** -- you type what to look for ("pallet,
14
+ cardboard box, forklift") and it searches for those names zero-shot, no
15
+ warehouse-specific training. Accuracy on warehouse imagery is NOT measured
16
+ here (no labelled real warehouse photos were available), so treat counts
17
+ as a starting point, not ground truth.
18
+
19
+ Why YOLOE rather than YOLO-World: both are open-vocabulary YOLO models, but
20
+ YOLO-World's text encoder (CLIP) is hosted on a CDN that could not be
21
+ reached from the development environment, so it could not be verified end
22
+ to end. YOLOE's text encoder (MobileCLIP) ships from the same GitHub
23
+ releases as the weights, and it was tested on a real photo before shipping.
24
+
25
+ Design notes
26
+ ------------
27
+ * Weights are downloaded on first use from the official Ultralytics GitHub
28
+ release assets and cached; nothing is bundled in the repo.
29
+ * `ultralytics`/`torch` are imported lazily, so if they're unavailable the
30
+ rest of the app still works and this tab shows a clear message.
31
+ * Example photos are fetched at runtime from Wikimedia Commons (openly
32
+ licensed) and shown with author + license. They are never bundled, so the
33
+ repo contains no third-party images. If the download fails, the user is
34
+ told to upload their own photo.
35
+ * Uploaded images are processed in memory and not stored.
36
+ """
37
+
38
+ import hashlib
39
+ import os
40
+ import tempfile
41
+ import threading
42
+ import time
43
+ from dataclasses import dataclass, field
44
+ from typing import Dict, List, Optional, Tuple
45
+
46
+ import requests
47
+ from PIL import Image, ImageDraw, ImageFont
48
+
49
+ WEIGHTS_DIR = os.environ.get("YOLO_WEIGHTS_DIR", os.path.join(tempfile.gettempdir(), "yolo_weights"))
50
+ EXAMPLES_DIR = os.path.join(tempfile.gettempdir(), "warehouse_examples")
51
+ ASSETS_BASE = "https://github.com/ultralytics/assets/releases/download/v8.3.0/"
52
+ USER_AGENT = "SmartWarehouseAIDemo/1.0 (portfolio demo; Hugging Face Space)"
53
+ MAX_SIDE = 1600 # downscale very large uploads before inference (CPU friendliness)
54
+
55
+ MODEL_CHOICES = {
56
+ "YOLOv8n (fastest)": "yolov8n.pt",
57
+ "YOLOv8s (more accurate)": "yolov8s.pt",
58
+ }
59
+ OPEN_VOCAB_WEIGHTS = "yoloe-11s-seg.pt"
60
+ TEXT_ENCODER_WEIGHTS = "mobileclip_blt.ts" # ~600 MB, downloaded once on first open-vocabulary use
61
+ DEFAULT_PROMPTS = "pallet, cardboard box, forklift, person, shelving rack"
62
+
63
+ # COCO classes that matter in a warehouse -> a plain-language group
64
+ WAREHOUSE_GROUPS = {
65
+ "person": "people",
66
+ "truck": "vehicles", "car": "vehicles", "bus": "vehicles",
67
+ "motorcycle": "vehicles", "bicycle": "vehicles", "train": "vehicles",
68
+ }
69
+
70
+ # Openly-licensed real photos from Wikimedia Commons (fetched at runtime, never bundled).
71
+ EXAMPLE_PHOTOS: Dict[str, dict] = {
72
+ "Pallet-rack warehouse with forklift": {
73
+ "file": "Modern_warehouse_with_pallet_rack_storage_system.jpg",
74
+ "author": "Axisadman", "license": "CC BY-SA 3.0",
75
+ "license_url": "https://creativecommons.org/licenses/by-sa/3.0/",
76
+ },
77
+ "Warehouse floor: pallets and forklift driver": {
78
+ "file": "Allentown_Project_042.jpg",
79
+ "author": "Chrisking", "license": "CC BY-SA 3.0",
80
+ "license_url": "https://creativecommons.org/licenses/by-sa/3.0/",
81
+ },
82
+ "Stacked CHEP pallets": {
83
+ "file": "576_CHEP_pallets.jpg",
84
+ "author": "train_photos (via Flickr)", "license": "CC BY-SA 2.0",
85
+ "license_url": "https://creativecommons.org/licenses/by-sa/2.0/",
86
+ },
87
+ }
88
+
89
+
90
+ @dataclass
91
+ class Detection:
92
+ label: str
93
+ conf: float
94
+ box: Tuple[int, int, int, int] # x1, y1, x2, y2
95
+
96
+
97
+ @dataclass
98
+ class RealDetectionResult:
99
+ annotated: Image.Image
100
+ detections: List[Detection]
101
+ counts: Dict[str, int]
102
+ inference_ms: float
103
+ model_name: str
104
+ notes: List[str] = field(default_factory=list)
105
+
106
+
107
+ # --------------------------------------------------------------------------
108
+ # Weights + model cache
109
+ # --------------------------------------------------------------------------
110
+ _MODEL_LOCK = threading.Lock()
111
+ _MODELS: Dict[str, object] = {}
112
+
113
+
114
+ def _ensure_weights(filename: str) -> str:
115
+ """Download an Ultralytics release asset once and cache it."""
116
+ os.makedirs(WEIGHTS_DIR, exist_ok=True)
117
+ path = os.path.join(WEIGHTS_DIR, filename)
118
+ if os.path.exists(path) and os.path.getsize(path) > 1_000_000:
119
+ return path
120
+ tmp = path + ".part"
121
+ with requests.get(ASSETS_BASE + filename, stream=True, timeout=60,
122
+ headers={"User-Agent": USER_AGENT}) as r:
123
+ r.raise_for_status()
124
+ with open(tmp, "wb") as f:
125
+ for chunk in r.iter_content(chunk_size=1 << 20):
126
+ f.write(chunk)
127
+ os.replace(tmp, path)
128
+ return path
129
+
130
+
131
+ def vision_stack_available() -> Tuple[bool, str]:
132
+ """Is the optional detector stack (ultralytics + torch) importable?"""
133
+ try:
134
+ import ultralytics # noqa: F401
135
+ return True, ""
136
+ except Exception as e: # noqa: BLE001
137
+ return False, f"{type(e).__name__}: {e}"
138
+
139
+
140
+ def _quiet_settings():
141
+ """Uploaded photos are user data: turn off Ultralytics' anonymous usage sync."""
142
+ try:
143
+ from ultralytics import settings
144
+ settings.update({"sync": False, "weights_dir": WEIGHTS_DIR})
145
+ except Exception: # noqa: BLE001
146
+ pass
147
+
148
+
149
+ def _get_yolo(weights_name: str):
150
+ with _MODEL_LOCK:
151
+ if weights_name not in _MODELS:
152
+ from ultralytics import YOLO
153
+ _quiet_settings()
154
+ _MODELS[weights_name] = YOLO(_ensure_weights(weights_name))
155
+ return _MODELS[weights_name]
156
+
157
+
158
+ def _get_open(prompts: Tuple[str, ...]):
159
+ """YOLOE with a given vocabulary. set_classes() re-encodes the text prompts,
160
+ so cache one model per distinct prompt set (a few at most)."""
161
+ key = "open::" + "|".join(prompts)
162
+ with _MODEL_LOCK:
163
+ if key not in _MODELS:
164
+ from ultralytics import YOLOE
165
+ _quiet_settings()
166
+ # make sure the ~600 MB text encoder is fetched from the official release
167
+ # assets into our cache dir, where ultralytics will look for it
168
+ _ensure_weights(TEXT_ENCODER_WEIGHTS)
169
+ model = YOLOE(_ensure_weights(OPEN_VOCAB_WEIGHTS))
170
+ model.set_classes(list(prompts), model.get_text_pe(list(prompts)))
171
+ _MODELS[key] = model
172
+ # keep the cache from growing without bound if people try many prompt sets
173
+ open_keys = [k for k in _MODELS if k.startswith("open::")]
174
+ for old in open_keys[:-3]:
175
+ _MODELS.pop(old, None)
176
+ return _MODELS[key]
177
+
178
+
179
+ # --------------------------------------------------------------------------
180
+ # Drawing
181
+ # --------------------------------------------------------------------------
182
+ _PALETTE = [(231, 76, 60), (52, 152, 219), (46, 204, 113), (241, 196, 15),
183
+ (155, 89, 182), (230, 126, 34), (26, 188, 156), (233, 30, 99)]
184
+
185
+
186
+ def _color_for(label: str) -> Tuple[int, int, int]:
187
+ h = int(hashlib.md5(label.encode()).hexdigest(), 16)
188
+ return _PALETTE[h % len(_PALETTE)]
189
+
190
+
191
+ def _font(size: int):
192
+ try:
193
+ return ImageFont.load_default(size=size) # Pillow >= 10.1
194
+ except TypeError:
195
+ return ImageFont.load_default()
196
+
197
+
198
+ def _annotate(image: Image.Image, dets: List[Detection]) -> Image.Image:
199
+ out = image.convert("RGB").copy()
200
+ draw = ImageDraw.Draw(out)
201
+ w, h = out.size
202
+ thick = max(2, round(max(w, h) / 400))
203
+ font = _font(max(12, round(max(w, h) / 55)))
204
+ for d in dets:
205
+ col = _color_for(d.label)
206
+ x1, y1, x2, y2 = d.box
207
+ draw.rectangle([x1, y1, x2, y2], outline=col, width=thick)
208
+ text = f"{d.label} {d.conf:.2f}"
209
+ try:
210
+ tb = draw.textbbox((x1, y1), text, font=font)
211
+ tw, th = tb[2] - tb[0], tb[3] - tb[1]
212
+ except Exception: # noqa: BLE001
213
+ tw, th = len(text) * 7, 12
214
+ ty = max(0, y1 - th - 4)
215
+ draw.rectangle([x1, ty, x1 + tw + 6, ty + th + 4], fill=col)
216
+ draw.text((x1 + 3, ty + 1), text, fill=(255, 255, 255), font=font)
217
+ return out
218
+
219
+
220
+ def _prepare(image: Image.Image) -> Image.Image:
221
+ img = image.convert("RGB")
222
+ if max(img.size) > MAX_SIDE:
223
+ img = img.copy()
224
+ img.thumbnail((MAX_SIDE, MAX_SIDE))
225
+ return img
226
+
227
+
228
+ def parse_prompts(text: str, limit: int = 12) -> Tuple[str, ...]:
229
+ items = []
230
+ for part in (text or "").replace("\n", ",").split(","):
231
+ p = part.strip().lower()
232
+ if p and p not in items:
233
+ items.append(p[:40])
234
+ return tuple(items[:limit])
235
+
236
+
237
+ # --------------------------------------------------------------------------
238
+ # Detection
239
+ # --------------------------------------------------------------------------
240
+ def _run(model, img: Image.Image, conf: float, iou: float = 0.7, names_override=None) -> List[Detection]:
241
+ results = model.predict(img, conf=conf, iou=iou, imgsz=640, verbose=False, device="cpu")
242
+ res = results[0]
243
+ names = names_override or res.names
244
+ dets: List[Detection] = []
245
+ if res.boxes is None:
246
+ return dets
247
+ for box in res.boxes:
248
+ cls_id = int(box.cls[0])
249
+ label = names[cls_id] if isinstance(names, (list, tuple)) else names.get(cls_id, str(cls_id))
250
+ x1, y1, x2, y2 = [int(round(v)) for v in box.xyxy[0].tolist()]
251
+ dets.append(Detection(label=str(label), conf=float(box.conf[0]), box=(x1, y1, x2, y2)))
252
+ dets.sort(key=lambda d: -d.conf)
253
+ return dets
254
+
255
+
256
+ def detect_real(image: Image.Image, mode: str = "standard", model_choice: str = "YOLOv8n (fastest)",
257
+ prompts_text: str = DEFAULT_PROMPTS, conf: float = 0.25, iou: float = 0.7) -> RealDetectionResult:
258
+ """mode: "standard" (COCO YOLOv8) or "open" (YOLOE, text prompts).
259
+
260
+ `iou` is the overlap threshold for non-max suppression. Lower values merge
261
+ overlapping boxes (fewer duplicate detections of one object) but can merge
262
+ genuinely separate, touching objects such as adjacent pallets -- so it is a
263
+ user-facing control rather than a value tuned on a single test image."""
264
+ img = _prepare(image)
265
+ notes: List[str] = []
266
+ t0 = time.perf_counter()
267
+ if mode == "open":
268
+ prompts = parse_prompts(prompts_text)
269
+ if not prompts:
270
+ raise ValueError("Enter at least one thing to look for, e.g. 'pallet, forklift'.")
271
+ model = _get_open(prompts)
272
+ dets = _run(model, img, conf, iou)
273
+ model_name = f"YOLOE (open-vocabulary) — looking for: {', '.join(prompts)}"
274
+ notes.append("Open-vocabulary detection is zero-shot: no warehouse-specific training, "
275
+ "and its accuracy on warehouse photos has **not** been measured here.")
276
+ else:
277
+ weights = MODEL_CHOICES.get(model_choice, "yolov8n.pt")
278
+ model = _get_yolo(weights)
279
+ dets = _run(model, img, conf, iou)
280
+ model_name = f"{weights} (COCO-pretrained, 80 classes)"
281
+ notes.append("COCO has no *pallet*, *forklift*, *box* or *rack* class, so this mode "
282
+ "can't count them — it finds people and vehicles. Switch to open-vocabulary "
283
+ "mode to search for pallets by name.")
284
+ ms = (time.perf_counter() - t0) * 1000
285
+ counts: Dict[str, int] = {}
286
+ for d in dets:
287
+ counts[d.label] = counts.get(d.label, 0) + 1
288
+ return RealDetectionResult(_annotate(img, dets), dets, counts, ms, model_name, notes)
289
+
290
+
291
+ def summarize(result: RealDetectionResult) -> str:
292
+ if not result.detections:
293
+ body = "**No objects detected** at this confidence threshold — try lowering it."
294
+ else:
295
+ rows = "\n".join(f"| {k} | {v} |" for k, v in sorted(result.counts.items(), key=lambda kv: -kv[1]))
296
+ body = f"**{len(result.detections)} detections**\n\n| Class | Count |\n|---|---|\n{rows}"
297
+ groups: Dict[str, int] = {}
298
+ for lbl, n in result.counts.items():
299
+ g = WAREHOUSE_GROUPS.get(lbl)
300
+ if g:
301
+ groups[g] = groups.get(g, 0) + n
302
+ if groups.get("people"):
303
+ body += (f"\n\n👷 **{groups['people']} person/people in frame** — a production system would "
304
+ f"cross-check this against AGV / forklift lanes for safety alerts.")
305
+ meta = f"\n\n*Model:* {result.model_name} · *Inference:* {result.inference_ms:.0f} ms (measured live on this server)"
306
+ notes = "".join(f"\n\n> ℹ️ {n}" for n in result.notes)
307
+ return body + meta + notes
308
+
309
+
310
+ # --------------------------------------------------------------------------
311
+ # Attributed example photos (Wikimedia Commons, fetched at runtime)
312
+ # --------------------------------------------------------------------------
313
+ def attribution_md(name: str) -> str:
314
+ info = EXAMPLE_PHOTOS[name]
315
+ page = "https://commons.wikimedia.org/wiki/File:" + info["file"]
316
+ return (f"📷 *{name}* — by **{info['author']}**, via [Wikimedia Commons]({page}), "
317
+ f"licensed [{info['license']}]({info['license_url']}). The annotated output is a derivative "
318
+ f"work, shared under the same license.")
319
+
320
+
321
+ def fetch_example(name: str) -> Tuple[Optional[Image.Image], str]:
322
+ """Returns (image or None, markdown message)."""
323
+ if name not in EXAMPLE_PHOTOS:
324
+ return None, "Pick an example from the list, or upload your own photo."
325
+ info = EXAMPLE_PHOTOS[name]
326
+ os.makedirs(EXAMPLES_DIR, exist_ok=True)
327
+ path = os.path.join(EXAMPLES_DIR, info["file"])
328
+ try:
329
+ if not (os.path.exists(path) and os.path.getsize(path) > 10_000):
330
+ url = f"https://commons.wikimedia.org/wiki/Special:FilePath/{info['file']}?width=1280"
331
+ r = requests.get(url, timeout=30, headers={"User-Agent": USER_AGENT}, allow_redirects=True)
332
+ r.raise_for_status()
333
+ if "image" not in r.headers.get("Content-Type", ""):
334
+ raise ValueError(f"unexpected content type {r.headers.get('Content-Type')}")
335
+ with open(path, "wb") as f:
336
+ f.write(r.content)
337
+ img = Image.open(path).convert("RGB")
338
+ return img, attribution_md(name)
339
+ except Exception as e: # noqa: BLE001
340
+ return None, (f"⚠️ Couldn't download the example photo ({type(e).__name__}). "
341
+ f"This server may be offline — please upload your own warehouse photo instead.")