| # Project Context: High-Throughput Diamond & Jewelry Vision Pipeline | |
| ## Domain | |
| - **Application**: Industrial diamond processing (100,000+ daily videos) | |
| - **Problem**: Segmentation, masking, and tracking of refractive/transparent gemstone objects | |
| - **Input**: Sorted JPEG frame sequences (N-frame sequences from video capture) | |
| - **Output**: Temporally-consistent soft-edge masks for downstream QC pipelines | |
| ## Core Stack & Models | |
| - **Propagation Engine**: Meta SAM 3 — `VideoPredictor` API for multi-frame mask propagation + open-vocabulary text prompts | |
| - **Student Detector**: YOLOv11-seg — Teacher-Student distillation for real-time inference | |
| - **Temporal Smoothing**: `scipy.signal.savgol_filter` (Savitzky-Golay) applied per-frame mask coefficients | |
| - **Performance Layer**: TensorRT export + batch inference (target: ≤10ms/frame) | |
| ## Refraction Rules (Strict) | |
| 1. **Never use binary masks** for diamonds or gemstones — always use soft-edge / alpha matting masks | |
| 2. **Alpha channel preservation**: Output masks must retain transparency gradients (float32 alpha map, 0.0–1.0) | |
| 3. **Edge softness**: Apply Gaussian-weighted alpha blending at mask boundaries (sigma ≥ 2px) | |
| 4. **Background reconstruction**: Use inpainting (e.g., `cv2.inpaint`) to handle semi-transparent regions | |
| ## Temporal Consistency Rules | |
| 1. **Savitzky-Golay filtering** MUST be applied across frame mask sequences (window=5, polyorder=2) | |
| 2. **No hard jumps**: Mask IoU between consecutive frames must be ≥ 0.85 (flag frames below threshold) | |
| 3. **Propagation priority**: Prefer SAM 3 propagation over per-frame YOLO prediction for tracked sequences | |
| 4. **Anchor frames**: Every 15th frame is re-annotated as a keyframe to prevent drift | |
| ## Architecture Rules | |
| 1. TensorRT FP16 for all inference — no FP32 in production paths | |
| 2. Batch size ≥ 8 for YOLO inference; ≥ 4 for SAM 3 propagation | |
| 3. Frame loading via OpenCV (`cv2.VideoCapture`) in sorted JPEG mode | |
| 4. All preprocessing must be GPU-side (CUDA streams) | |
| 5. Use `torch.compile()` for Python-side model wrappers where supported | |
| ## Naming Conventions | |
| - Mask output files: `frame_{idx:06d}_mask_alpha.png` | |
| - Model checkpoints: `yolo11seg_diamond_v{version}.pt` / `sam3_diamond_v{version}.pth` | |
| - Dataset splits: `train/`, `val/`, `test/` under `data/` | |
| ## Quality Criteria | |
| - Precision / Recall for diamond facets: target ≥ 0.92 mAP@0.5 | |
| - Mask temporal SSIM: ≥ 0.95 across N-frame sequences | |
| - Throughput: ≥ 500 frames/sec on A100 80GB (batch mode) | |