TESTING.md

# SAM3 Testing Guide

## Overview

This guide covers two testing approaches for SAM3:

1. **Basic Inference Testing** - Quick API validation with sample images
2. **Metrics Evaluation** - Comprehensive performance analysis against CVAT ground truth

---

## 1. Basic Inference Testing

### Purpose

Quickly validate that the SAM3 endpoint is working and producing reasonable segmentation results.

### Test Infrastructure

The basic testing framework:
- Tests multiple images automatically
- Saves detailed JSON logs of requests and responses
- Generates visualizations with semi-transparent colored masks
- Stores all results in `.cache/test/inference/{image_name}/`

### Running Basic Tests

```bash
python3 scripts/test/test_inference_comprehensive.py
```

### Test Output Structure

For each test image, files are generated in `.cache/test/inference/{image_name}/`:

- `request.json` - Request metadata (timestamp, endpoint, classes)
- `response.json` - Response metadata (timestamp, status, results summary)
- `full_results.json` - Complete API response including base64 masks
- `original.jpg` - Original test image
- `visualization.png` - Original image with colored mask overlay
- `legend.png` - Legend showing class colors and coverage percentages
- `mask_{ClassName}.png` - Individual binary masks for each class

### Tested Classes

The endpoint is tested with these semantic classes:
- **Pothole** (Red overlay)
- **Road crack** (Yellow overlay)
- **Road** (Blue overlay)

### Recent Test Results

**Last run**: November 23, 2025

- **Total images**: 8
- **Successful**: 8/8 (100%)
- **Failed**: 0
- **Average response time**: ~1.5 seconds per image
- **Status**: All API calls returning HTTP 200 with valid masks

Test images include:
- `pothole_pexels_01.jpg`, `pothole_pexels_02.jpg`
- `road_damage_01.jpg`
- `road_pexels_01.jpg`, `road_pexels_02.jpg`, `road_pexels_03.jpg`
- `road_unsplash_01.jpg`
- `test.jpg`

Results stored in `.cache/test/inference/summary.json`

### Adding More Test Images

Test images should be placed in `assets/test_images/`. To expand the test suite:

1. **Download from Public Datasets**:
   - [Pothole Detection Dataset](https://github.com/jaygala24/pothole-detection/releases/download/v1.0.0/Pothole.Dataset.IVCNZ.zip) (1,243 images)
   - [RDD2022 Dataset](https://github.com/sekilab/RoadDamageDetector) (47,420 images from 6 countries)
   - [Roboflow Pothole Dataset](https://public.roboflow.com/object-detection/pothole/)

2. **Extract Sample Images**: Select diverse examples showing potholes, cracks, and clean roads

3. **Place in Test Directory**: Copy to `assets/test_images/`

---

## 2. Metrics Evaluation System

### Purpose

Comprehensive quantitative evaluation of SAM3 performance against ground truth annotations from CVAT.

### What It Measures

- **mAP (mean Average Precision)**: Detection accuracy across all confidence thresholds
- **mAR (mean Average Recall)**: Coverage of ground truth instances
- **IoU metrics**: Intersection over Union at multiple thresholds (0%, 25%, 50%, 75%)
- **Confusion matrices**: Class prediction accuracy patterns
- **Per-class statistics**: Precision, recall, F1-score for each damage type

### Running Metrics Evaluation

```bash
cd metrics_evaluation
python run_evaluation.py
```

**Options**:
```bash
# Force re-download from CVAT (ignore cache)
python run_evaluation.py --force-download

# Force re-run inference (ignore cached predictions)
python run_evaluation.py --force-inference

# Skip inference step (use existing predictions)
python run_evaluation.py --skip-inference

# Generate visual comparisons
python run_evaluation.py --visualize
```

### Dataset

Evaluates on **150 annotated images** from CVAT:
- **50 images** with "Fissure" (road cracks)
- **50 images** with "Nid de poule" (potholes)
- **50 images** with road surface

Source: Logiroad CVAT organization, AI training project

### Output Structure

```
.cache/test/metrics/
├── Fissure/
│   └── {image_name}/
│       ├── image.jpg
│       ├── ground_truth/
│       │   ├── mask_Fissure_0.png
│       │   └── metadata.json
│       └── inference/
│           ├── mask_Fissure_0.png
│           └── metadata.json
├── Nid de poule/
├── Road/
├── metrics_summary.txt        # Human-readable results
├── metrics_detailed.json      # Complete metrics data
└── evaluation_log.txt         # Execution trace
```

### Execution Time

- Image download: ~5-10 minutes (150 images)
- SAM3 inference: ~5-10 minutes (~2s per image)
- Metrics computation: ~1 minute
- **Total**: ~15-20 minutes for full evaluation

### Configuration

Edit `metrics_evaluation/config/config.json` to:
- Change CVAT project or organization
- Adjust number of images per class
- Modify IoU thresholds
- Update SAM3 endpoint URL

CVAT credentials must be in `.env` at project root.

---

## Cache Directory

All test results are stored in `.cache/` (git-ignored):
- Review results without cluttering the repository
- Compare results across different test runs
- Debug segmentation quality issues
- Resume interrupted evaluations

---

## Quality Validation Checklist

Before accepting test results:

**Basic Tests**:
- [ ] All test images processed successfully
- [ ] Masks generated for all requested classes
- [ ] Response times reasonable (< 3s per image)
- [ ] Visualizations show plausible segmentations

**Metrics Evaluation**:
- [ ] 150 images downloaded from CVAT
- [ ] Ground truth masks not empty
- [ ] SAM3 inference completed for all images
- [ ] Metrics within reasonable ranges (0-100%)
- [ ] Confusion matrices show sensible patterns
- [ ] Per-class F1 scores above baseline

---

## Troubleshooting

### Basic Inference Issues

**Endpoint not responding**:
- Check endpoint URL in test script
- Verify endpoint is running (use `curl` or browser)
- Check network connectivity

**Empty or invalid masks**:
- Review class names match model expectations
- Check image format (should be JPEG/PNG)
- Verify base64 encoding/decoding

### Metrics Evaluation Issues

**CVAT connection fails**:
- Check `.env` credentials
- Verify CVAT organization name
- Test CVAT web access

**No images found**:
- Check project filter in `config.json`
- Verify labels exist in CVAT
- Ensure images have annotations

**Metrics seem incorrect**:
- Inspect confusion matrices
- Review sample visualizations
- Check ground truth quality in CVAT
- Verify mask format (PNG-L, 8-bit grayscale)

---

## Next Steps

1. **Run basic tests** to validate API connectivity
2. **Review visualizations** to assess segmentation quality
3. **Run metrics evaluation** for quantitative performance
4. **Analyze confusion matrices** to identify systematic errors
5. **Iterate on model/prompts** based on metrics feedback

For detailed metrics evaluation documentation, see `metrics_evaluation/README.md`.