| --- |
| title: BOM Pattern Detection |
| emoji: "⬜" |
| colorFrom: blue |
| colorTo: green |
| sdk: docker |
| pinned: false |
| app_port: 7860 |
| --- |
| |
| # Zero-Shot Pattern Detection for Engineering BOM Drawings |
|
|
|  |
|  |
|  |
|  |
|  |
|
|
| ## Overview |
|
|
| Find every occurrence of a given component symbol inside large engineering BOM drawings — with **zero training data**. This is a Sotatek AI/Computer Vision assessment project that implements a complete **zero-shot pattern detection pipeline** combining three intelligence layers: |
|
|
| 1. **Classical NCC multi-scale matching** — fast CPU-based candidate proposal |
| 2. **DINOv2 ViT-S/14 zero-shot verification** — self-supervised semantic filtering |
| 3. **Optional Qwen2-VL-2B VLM semantic classifier** — borderline confidence refinement |
|
|
| **No fine-tuning or labelled data needed.** Any pattern template works at inference time. |
|
|
| --- |
|
|
| ## Results (Test Drawing 4 + Zigzag Resistor Template) |
|
|
| | Metric | Value | |
| |--------|-------| |
| | **GT Boxes Detected** | 22/24 (91.7% recall) | |
| | **False Positives** | 0 | |
| | **Total Detections** | 22 | |
| | **Runtime (GPU RTX 3060)** | ~25 seconds | |
| | **Unit Tests** | 10/10 passing ✓ | |
|
|
| --- |
|
|
| ## Pipeline Architecture |
|
|
| ``` |
| ┌──────────────┐ ┌──────────────┐ |
| │ Pattern IMG │ │ Drawing IMG │ |
| └──────┬───────┘ └───────┬──────┘ |
| │ │ |
| └────────────┬────────────┘ |
| │ |
| ┌───────▼────────┐ |
| │ Stage 0: │ |
| │ Preprocess │ |
| │ (binarize, │ |
| │ denoise) │ |
| └────────┬───────┘ |
| │ < 0.5 s |
| ┌───────────▼────────────┐ |
| │ Stage 1: NCC Matching │ |
| │ (multi-scale, ±10°,90°)│ |
| │ Candidate Proposal │ |
| │ │ |
| │ CPU: 30-60s | GPU: 15s │ |
| └───────────┬────────────┘ |
| │ ~200-400 candidates |
| ┌───────────▼────────────┐ |
| │ Stage 2: DINOv2 Verify │ |
| │ (cosine similarity) │ |
| │ Zero-shot (no ft) │ |
| │ │ |
| │ GPU: 2-10s │ |
| └───────────┬────────────┘ |
| │ ~50-100 candidates |
| ┌───────────▼───────────┐ |
| │ Stage 2b: Filters │ |
| │ • Wire-leads │ |
| │ • Chamfer distance │ |
| │ • Structural checks │ |
| │ • NMS │ |
| │ • Gap filter │ |
| │ │ |
| │ < 1 s │ |
| └───────────┬───────────┘ |
| │ ~20-30 candidates |
| [if use_vlm=True] |
| ┌───────────▼───────────┐ |
| │ Stage 3: VLM Filter │ |
| │ (Qwen2-VL-2B) │ |
| │ Open-classification │ |
| │ Borderline only │ |
| │ │ |
| │ ~0.4 s/crop │ |
| └───────────┬───────────┘ |
| │ |
| ┌────────▼─────────┐ |
| │ Output: │ |
| │ BBoxes + Scores │ |
| │ JSON │ |
| └──────────────────┘ |
| ``` |
|
|
| --- |
|
|
| ## Installation |
|
|
| ### Requirements |
| - Python 3.11+ |
| - CUDA 11.8+ (optional, for GPU acceleration) |
| - 4GB+ RAM (8GB+ recommended for VLM) |
|
|
| ### Setup |
|
|
| ```bash |
| # Clone repo |
| git clone https://github.com/YOUR_USERNAME/pattern-detection-bom.git |
| cd pattern-detection-bom |
| |
| # Create virtual environment |
| python -m venv venv |
| source venv/bin/activate # Windows: venv\Scripts\activate |
| |
| # Install dependencies |
| pip install -r requirements.txt |
| ``` |
|
|
| **Model Weights** download automatically on first run: |
| - **DINOv2 ViT-S/14** (~86 MB) — via `torch.hub` |
| - **Qwen2-VL-2B** (~4.5 GB, optional) — via HuggingFace Hub (lazy-loaded when `use_vlm=True`) |
|
|
| --- |
|
|
| ## Quick Start |
|
|
| ### Python API |
|
|
| ```python |
| from src.pipeline import PatternDetectionPipeline |
| |
| # Initialize (GPU auto-detected) |
| pipe = PatternDetectionPipeline() |
| |
| # Run detection |
| result = pipe.detect_auto( |
| pattern_path="template.png", |
| drawing_path="schematic.png", |
| return_visualization=True |
| ) |
| |
| # Access results |
| print(f"Found {result['total_detections']} instances") |
| for det in result['detections']: |
| bbox = det['bbox'] |
| print(f" ({bbox['x']}, {bbox['y']}) {bbox['w']}×{bbox['h']} | " |
| f"conf={det['confidence']:.3f} | dino={det['dino_score']:.3f}") |
| ``` |
|
|
| ### Web UI (FastAPI) |
|
|
| ```bash |
| # Start server |
| python app/web/server.py |
| |
| # Open browser |
| # http://localhost:8000 |
| ``` |
|
|
| **UI Features:** |
| - Drag-and-drop image upload |
| - Auto-tune mode + manual threshold sliders |
| - Real-time visualization with bounding boxes |
| - Download results (PNG + CSV) |
| - **About page** with system specification link |
| - **VLM toggle** for borderline confidence filtering |
|
|
| ### Docker (HuggingFace Spaces) |
|
|
| ```bash |
| docker build -t bom-detector . |
| docker run -p 7860:7860 bom-detector |
| ``` |
|
|
| --- |
|
|
| ## Output Format |
|
|
| ```json |
| { |
| "success": true, |
| "total_detections": 3, |
| "elapsed": 24.52, |
| "detections": [ |
| { |
| "bbox": { |
| "x": 142, |
| "y": 310, |
| "w": 64, |
| "h": 32 |
| }, |
| "confidence": 0.905, |
| "ncc_score": 0.823, |
| "dino_score": 0.897, |
| "scale": 1.05, |
| "angle": 0.0 |
| } |
| ], |
| "visualization": "base64_png_string" |
| } |
| ``` |
|
|
| --- |
|
|
| ## Configuration |
|
|
| ### Pipeline Parameters |
|
|
| | Parameter | Default | Description | |
| |-----------|---------|-------------| |
| | `ncc_threshold` | `0.55` / `0.47` | NCC gate (strict/relaxed) | |
| | `cosine_threshold` | `0.84` | DINOv2 cosine similarity minimum | |
| | `final_nms_iou` | `0.40` | Final NMS IoU threshold | |
| | `use_vlm` | `False` | Enable Qwen2-VL stage | |
| | `vlm_keep_min_conf` | `0.75` | Skip VLM for confidence ≥ this | |
| | `vlm_reject_only` | `True` | Blacklist mode (recommended) | |
| | `vlm_recall_boost` | `auto` | Relax gates when VLM on | |
|
|
| ### Python Config Example |
|
|
| ```python |
| config = { |
| "cosine_threshold": 0.84, |
| "final_nms_iou": 0.40, |
| "use_vlm": True, |
| "vlm_symbol_name": "a zigzag resistor", |
| "vlm_keep_min_conf": 0.78, |
| "vlm_reject_only": True, |
| } |
| |
| pipe = PatternDetectionPipeline(config=config) |
| result = pipe.detect_auto("pattern.png", "drawing.png") |
| ``` |
|
|
| --- |
|
|
| ## Design Choices |
|
|
| ### Why NCC + DINOv2? |
|
|
| - **NCC alone** is fast but brittle (sensitive to scale/noise) |
| - **DINOv2 dense sliding-window** is accurate but prohibitively slow |
| - **Hybrid approach** = fast proposal (NCC) + accurate verification (DINOv2) |
|
|
| ### Why DINOv2 over CLIP? |
|
|
| - **CLIP image-to-image** offers no better FP separation than DINOv2 on line-art |
| - **CLIP text-guided** breaks zero-shot requirement (need to know class names) |
| - **DINOv2 self-supervised** generalizes well to technical drawings despite natural-image training |
|
|
| ### Why VLM open-classification instead of yes/no? |
|
|
| - **Yes/no prompting** causes 100% agreement bias with small VLMs |
| - **Open-classification** forces the model to choose from a fixed vocabulary |
| - **Blacklist mode** (`vlm_reject_only=True`) avoids over-aggressive whitelisting |
|
|
| --- |
|
|
| ## Stages in Detail |
|
|
| ### Stage 1: NCC Multi-scale Matching |
| - Scales: 0.70×–1.80× (adaptive per template) |
| - Angles: ±10° (standard) + 0°/90° (complex templates) |
| - NCC threshold: 0.55 (strict) / 0.47 (relaxed) |
| - Outputs: 30–400 candidates per drawing |
|
|
| ### Stage 2: DINOv2 Zero-Shot Verification |
| - Model: `dinov2_vits14` (21M params, 384-D embeddings) |
| - Approach: cosine similarity on center-crop + full-crop (max) |
| - Derotation: 90° candidates derotated to horizontal for comparison |
| - Outputs: ~50–100 verified candidates |
|
|
| ### Stage 2b: Structural Filters |
| - **Wire-leads**: Real components have visible leads (probe left/right/top/bottom) |
| - **Chamfer distance**: Edge alignment between template and crop (max 5.0) |
| - **Neighborhood complexity**: Component must sit in sparse region |
| - **Junction dots / Rect integrity**: Reject partial matches, L-corners |
| - **Confidence gap**: Drop low-confidence tail if bimodal distribution detected |
| - **Final NMS**: Containment-aware suppression (IoU threshold 0.4) |
|
|
| ### Stage 3: VLM Semantic Filter (Optional) |
| - Model: `Qwen/Qwen2-VL-2B-Instruct` (~5GB, lazy-loaded) |
| - Classes: resistor, inductor, capacitor, diode, crystal, transistor, op-amp, logic-gate, wire-junction, other |
| - Mode: **Blacklist** (reject only known FPs, trust unknowns) |
| - Target: Borderline candidates only (confidence < 0.75) |
|
|
| --- |
|
|
| ## Documentation |
|
|
| **For detailed system specification, see:** |
| - [System Specification (HTML)](design_spec/system_spec.html) — complete architecture, algorithms, experimental results |
| - [Model Survey](design_spec/model_survey.md) — DINOv2 vs CLIP, VLM yes/no vs open-classification experiments |
|
|
| **Web UI About page** links to full specification (click **About** → **Mở đặc tả hệ thống**). |
|
|
| --- |
|
|
| ## Testing |
|
|
| ```bash |
| # Run all tests |
| pytest tests/ -v |
| |
| # Run specific test |
| pytest tests/test_pipeline.py::test_detect_auto -v |
| |
| # Test coverage |
| pytest tests/ --cov=src --cov-report=html |
| ``` |
|
|
| **Test Results:** 10/10 passing ✓ |
| - Simple template detection |
| - Complex template detection |
| - Output format validation |
| - Config override tests |
| - VLM toggle tests |
|
|
| --- |
|
|
| ## Project Structure |
|
|
| ``` |
| pattern-detection-bom/ |
| ├── src/ |
| │ ├── pipeline.py # Main orchestrator |
| │ ├── preprocessor.py # Image binarization, denoise |
| │ ├── ncc_matcher.py # NCC multi-scale matching |
| │ ├── dino_verifier.py # DINOv2 zero-shot verification |
| │ ├── postprocessor.py # Structural filters, NMS |
| │ ├── vlm_verifier.py # Qwen2-VL semantic filter (optional) |
| │ └── dino_dense_matcher.py # DINOv2 dense sliding window (fallback) |
| │ |
| ├── app/web/ |
| │ ├── server.py # FastAPI server |
| │ ├── index.html # SPA frontend |
| │ └── static/ |
| │ ├── css/style.css # UI styling |
| │ └── js/app.js # Frontend logic (page nav, detection) |
| │ |
| ├── design_spec/ |
| │ ├── system_spec.html # Complete system documentation (13 sections) |
| │ ├── model_survey.md # Model comparison experiments |
| │ └── design_decisions.md # Architecture rationale |
| │ |
| ├── tests/ |
| │ ├── test_pipeline.py # 10 unit tests |
| │ └── conftest.py # Pytest fixtures |
| │ |
| ├── Dockerfile # HuggingFace Spaces deployment |
| ├── requirements.txt # Python dependencies |
| ├── .gitignore # Git ignore rules |
| └── README.md # This file |
| ``` |
|
|
| --- |
|
|
| ## Performance |
|
|
| | Metric | Value | |
| |--------|-------| |
| | **Speed (GPU RTX 3060 12GB)** | 20–30 seconds per A3 drawing | |
| | **Speed (CPU i7-10700)** | 60–120 seconds per A3 drawing | |
| | **Memory (GPU)** | 4.5 GB (DINOv2) + 4.5 GB (optional VLM) | |
| | **Memory (CPU)** | 2 GB | |
| | **Recall (test drawing 4)** | 91.7% (22/24 GT boxes) | |
| | **False Positives** | 0 (after DINOv2 + structural filters) | |
|
|
| --- |
|
|
| ## Known Limitations |
|
|
| 1. **Boundary resistors** — Symbols at image edges may be suppressed by NMS |
| 2. **Resistor in framing box** — When a symbol sits inside a dense bounding frame, DINOv2 score collides with FP diodes (irreducible without VLM) |
| 3. **Heavy rotation (> 15°)** — Only ±10° and 90° supported |
| 4. **VLM labels** — Qwen2-VL-2B individually noisy at borderline (effective at population level) |
| 5. **Single-draw processing** — No batch inference yet |
|
|
| --- |
|
|
| ## Development |
|
|
| ### Running with custom config |
|
|
| ```python |
| from src.pipeline import PatternDetectionPipeline |
| |
| pipe = PatternDetectionPipeline(config={ |
| "cosine_threshold": 0.88, # stricter DINOv2 |
| "use_vlm": True, |
| "vlm_symbol_name": "a resistor", |
| }) |
| ``` |
|
|
| ### Adding a new structural filter |
|
|
| Edit `src/postprocessor.py`, add method to `Postprocessor` class following the pattern of `filter_wire_leads()`. |
|
|
| --- |
|
|
| ## License |
|
|
| MIT License — see [LICENSE](LICENSE) for details. |
|
|
| --- |
|
|
| ## Author |
|
|
| **Bảo Duy** — [zestdapoet@gmail.com](mailto:zestdapoet@gmail.com) |
|
|
| - **GitHub:** [@DuyhocAI](https://github.com/DuyhocAI) |
| - **Project:** Sotatek AI/Computer Vision Engineer Assessment |
| - **Duration:** 96 hours |
| - **Submission:** June 1, 2026 |
|
|
| See [AUTHORS.md](AUTHORS.md) for full credits and framework acknowledgments. |
|
|
| --- |
|
|
| ## Acknowledgments |
|
|
| - **DINOv2** — Facebook AI ([Oquab et al., 2023](https://arxiv.org/abs/2304.07193)) |
| - **Qwen2-VL** — Alibaba Qwen Team |
| - **FastAPI** — Modern web framework for Python |
|
|
