metadata
title: BOM Pattern Detection
emoji: ⬜
colorFrom: blue
colorTo: green
sdk: docker
pinned: false
app_port: 7860
Zero-Shot Pattern Detection for Engineering BOM Drawings
Overview
Find every occurrence of a given component symbol inside large engineering BOM drawings — with zero training data. This is a Sotatek AI/Computer Vision assessment project that implements a complete zero-shot pattern detection pipeline combining three intelligence layers:
- Classical NCC multi-scale matching — fast CPU-based candidate proposal
- DINOv2 ViT-S/14 zero-shot verification — self-supervised semantic filtering
- Optional Qwen2-VL-2B VLM semantic classifier — borderline confidence refinement
No fine-tuning or labelled data needed. Any pattern template works at inference time.
Results (Test Drawing 4 + Zigzag Resistor Template)
| Metric | Value |
|---|---|
| GT Boxes Detected | 22/24 (91.7% recall) |
| False Positives | 0 |
| Total Detections | 22 |
| Runtime (GPU RTX 3060) | ~25 seconds |
| Unit Tests | 10/10 passing ✓ |
Pipeline Architecture
┌──────────────┐ ┌──────────────┐
│ Pattern IMG │ │ Drawing IMG │
└──────┬───────┘ └───────┬──────┘
│ │
└────────────┬────────────┘
│
┌───────▼────────┐
│ Stage 0: │
│ Preprocess │
│ (binarize, │
│ denoise) │
└────────┬───────┘
│ < 0.5 s
┌───────────▼────────────┐
│ Stage 1: NCC Matching │
│ (multi-scale, ±10°,90°)│
│ Candidate Proposal │
│ │
│ CPU: 30-60s | GPU: 15s │
└───────────┬────────────┘
│ ~200-400 candidates
┌───────────▼────────────┐
│ Stage 2: DINOv2 Verify │
│ (cosine similarity) │
│ Zero-shot (no ft) │
│ │
│ GPU: 2-10s │
└───────────┬────────────┘
│ ~50-100 candidates
┌───────────▼───────────┐
│ Stage 2b: Filters │
│ • Wire-leads │
│ • Chamfer distance │
│ • Structural checks │
│ • NMS │
│ • Gap filter │
│ │
│ < 1 s │
└───────────┬───────────┘
│ ~20-30 candidates
[if use_vlm=True]
┌───────────▼───────────┐
│ Stage 3: VLM Filter │
│ (Qwen2-VL-2B) │
│ Open-classification │
│ Borderline only │
│ │
│ ~0.4 s/crop │
└───────────┬───────────┘
│
┌────────▼─────────┐
│ Output: │
│ BBoxes + Scores │
│ JSON │
└──────────────────┘
Installation
Requirements
- Python 3.11+
- CUDA 11.8+ (optional, for GPU acceleration)
- 4GB+ RAM (8GB+ recommended for VLM)
Setup
# Clone repo
git clone https://github.com/YOUR_USERNAME/pattern-detection-bom.git
cd pattern-detection-bom
# Create virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
Model Weights download automatically on first run:
- DINOv2 ViT-S/14 (~86 MB) — via
torch.hub - Qwen2-VL-2B (~4.5 GB, optional) — via HuggingFace Hub (lazy-loaded when
use_vlm=True)
Quick Start
Python API
from src.pipeline import PatternDetectionPipeline
# Initialize (GPU auto-detected)
pipe = PatternDetectionPipeline()
# Run detection
result = pipe.detect_auto(
pattern_path="template.png",
drawing_path="schematic.png",
return_visualization=True
)
# Access results
print(f"Found {result['total_detections']} instances")
for det in result['detections']:
bbox = det['bbox']
print(f" ({bbox['x']}, {bbox['y']}) {bbox['w']}×{bbox['h']} | "
f"conf={det['confidence']:.3f} | dino={det['dino_score']:.3f}")
Web UI (FastAPI)
# Start server
python app/web/server.py
# Open browser
# http://localhost:8000
UI Features:
- Drag-and-drop image upload
- Auto-tune mode + manual threshold sliders
- Real-time visualization with bounding boxes
- Download results (PNG + CSV)
- About page with system specification link
- VLM toggle for borderline confidence filtering
Docker (HuggingFace Spaces)
docker build -t bom-detector .
docker run -p 7860:7860 bom-detector
Output Format
{
"success": true,
"total_detections": 3,
"elapsed": 24.52,
"detections": [
{
"bbox": {
"x": 142,
"y": 310,
"w": 64,
"h": 32
},
"confidence": 0.905,
"ncc_score": 0.823,
"dino_score": 0.897,
"scale": 1.05,
"angle": 0.0
}
],
"visualization": "base64_png_string"
}
Configuration
Pipeline Parameters
| Parameter | Default | Description |
|---|---|---|
ncc_threshold |
0.55 / 0.47 |
NCC gate (strict/relaxed) |
cosine_threshold |
0.84 |
DINOv2 cosine similarity minimum |
final_nms_iou |
0.40 |
Final NMS IoU threshold |
use_vlm |
False |
Enable Qwen2-VL stage |
vlm_keep_min_conf |
0.75 |
Skip VLM for confidence ≥ this |
vlm_reject_only |
True |
Blacklist mode (recommended) |
vlm_recall_boost |
auto |
Relax gates when VLM on |
Python Config Example
config = {
"cosine_threshold": 0.84,
"final_nms_iou": 0.40,
"use_vlm": True,
"vlm_symbol_name": "a zigzag resistor",
"vlm_keep_min_conf": 0.78,
"vlm_reject_only": True,
}
pipe = PatternDetectionPipeline(config=config)
result = pipe.detect_auto("pattern.png", "drawing.png")
Design Choices
Why NCC + DINOv2?
- NCC alone is fast but brittle (sensitive to scale/noise)
- DINOv2 dense sliding-window is accurate but prohibitively slow
- Hybrid approach = fast proposal (NCC) + accurate verification (DINOv2)
Why DINOv2 over CLIP?
- CLIP image-to-image offers no better FP separation than DINOv2 on line-art
- CLIP text-guided breaks zero-shot requirement (need to know class names)
- DINOv2 self-supervised generalizes well to technical drawings despite natural-image training
Why VLM open-classification instead of yes/no?
- Yes/no prompting causes 100% agreement bias with small VLMs
- Open-classification forces the model to choose from a fixed vocabulary
- Blacklist mode (
vlm_reject_only=True) avoids over-aggressive whitelisting
Stages in Detail
Stage 1: NCC Multi-scale Matching
- Scales: 0.70×–1.80× (adaptive per template)
- Angles: ±10° (standard) + 0°/90° (complex templates)
- NCC threshold: 0.55 (strict) / 0.47 (relaxed)
- Outputs: 30–400 candidates per drawing
Stage 2: DINOv2 Zero-Shot Verification
- Model:
dinov2_vits14(21M params, 384-D embeddings) - Approach: cosine similarity on center-crop + full-crop (max)
- Derotation: 90° candidates derotated to horizontal for comparison
- Outputs: ~50–100 verified candidates
Stage 2b: Structural Filters
- Wire-leads: Real components have visible leads (probe left/right/top/bottom)
- Chamfer distance: Edge alignment between template and crop (max 5.0)
- Neighborhood complexity: Component must sit in sparse region
- Junction dots / Rect integrity: Reject partial matches, L-corners
- Confidence gap: Drop low-confidence tail if bimodal distribution detected
- Final NMS: Containment-aware suppression (IoU threshold 0.4)
Stage 3: VLM Semantic Filter (Optional)
- Model:
Qwen/Qwen2-VL-2B-Instruct(~5GB, lazy-loaded) - Classes: resistor, inductor, capacitor, diode, crystal, transistor, op-amp, logic-gate, wire-junction, other
- Mode: Blacklist (reject only known FPs, trust unknowns)
- Target: Borderline candidates only (confidence < 0.75)
Documentation
For detailed system specification, see:
- System Specification (HTML) — complete architecture, algorithms, experimental results
- Model Survey — DINOv2 vs CLIP, VLM yes/no vs open-classification experiments
Web UI About page links to full specification (click About → Mở đặc tả hệ thống).
Testing
# Run all tests
pytest tests/ -v
# Run specific test
pytest tests/test_pipeline.py::test_detect_auto -v
# Test coverage
pytest tests/ --cov=src --cov-report=html
Test Results: 10/10 passing ✓
- Simple template detection
- Complex template detection
- Output format validation
- Config override tests
- VLM toggle tests
Project Structure
pattern-detection-bom/
├── src/
│ ├── pipeline.py # Main orchestrator
│ ├── preprocessor.py # Image binarization, denoise
│ ├── ncc_matcher.py # NCC multi-scale matching
│ ├── dino_verifier.py # DINOv2 zero-shot verification
│ ├── postprocessor.py # Structural filters, NMS
│ ├── vlm_verifier.py # Qwen2-VL semantic filter (optional)
│ └── dino_dense_matcher.py # DINOv2 dense sliding window (fallback)
│
├── app/web/
│ ├── server.py # FastAPI server
│ ├── index.html # SPA frontend
│ └── static/
│ ├── css/style.css # UI styling
│ └── js/app.js # Frontend logic (page nav, detection)
│
├── design_spec/
│ ├── system_spec.html # Complete system documentation (13 sections)
│ ├── model_survey.md # Model comparison experiments
│ └── design_decisions.md # Architecture rationale
│
├── tests/
│ ├── test_pipeline.py # 10 unit tests
│ └── conftest.py # Pytest fixtures
│
├── Dockerfile # HuggingFace Spaces deployment
├── requirements.txt # Python dependencies
├── .gitignore # Git ignore rules
└── README.md # This file
Performance
| Metric | Value |
|---|---|
| Speed (GPU RTX 3060 12GB) | 20–30 seconds per A3 drawing |
| Speed (CPU i7-10700) | 60–120 seconds per A3 drawing |
| Memory (GPU) | 4.5 GB (DINOv2) + 4.5 GB (optional VLM) |
| Memory (CPU) | 2 GB |
| Recall (test drawing 4) | 91.7% (22/24 GT boxes) |
| False Positives | 0 (after DINOv2 + structural filters) |
Known Limitations
- Boundary resistors — Symbols at image edges may be suppressed by NMS
- Resistor in framing box — When a symbol sits inside a dense bounding frame, DINOv2 score collides with FP diodes (irreducible without VLM)
- Heavy rotation (> 15°) — Only ±10° and 90° supported
- VLM labels — Qwen2-VL-2B individually noisy at borderline (effective at population level)
- Single-draw processing — No batch inference yet
Development
Running with custom config
from src.pipeline import PatternDetectionPipeline
pipe = PatternDetectionPipeline(config={
"cosine_threshold": 0.88, # stricter DINOv2
"use_vlm": True,
"vlm_symbol_name": "a resistor",
})
Adding a new structural filter
Edit src/postprocessor.py, add method to Postprocessor class following the pattern of filter_wire_leads().
License
MIT License — see LICENSE for details.
Author
Bảo Duy — zestdapoet@gmail.com
- GitHub: @DuyhocAI
- Project: Sotatek AI/Computer Vision Engineer Assessment
- Duration: 96 hours
- Submission: June 1, 2026
See AUTHORS.md for full credits and framework acknowledgments.
Acknowledgments
- DINOv2 — Facebook AI (Oquab et al., 2023)
- Qwen2-VL — Alibaba Qwen Team
- FastAPI — Modern web framework for Python