README.md

---
license: agpl-3.0
tags:
  - image-classification
  - onnx
  - pytorch
  - omr
  - checkbox-detection
  - plom
pipeline_tag: image-classification
---

# OMR Checkbox Classifier

A lightweight binary CNN that classifies scanned exam answer-sheet checkboxes as **filled** or **empty**. Trained on synthetic crops generated from PDF answer-sheet templates, with augmentation to simulate real scan noise, skew, and marking styles.

Developed for the [Plom Project](https://gitlab.com/plom/plom) — an open-source platform for automated grading of paper exams.

- **Developer**: Chenfeng Xu, Undergraduate Research Assistant, UBC
- **Supervisors**: Prof. Andrew Rechnitzer & Prof. Colin B. MacDonald
- **License**: AGPL-3.0 (to comply with Plom Project requirements)

---

## Model Details

| Property | Value |
|---|---|
| **Architecture** | SmallOMRNet (custom lightweight CNN) |
| **Input** | 1 × 64 × 64 grayscale image |
| **Output** | Scalar sigmoid probability of checkbox being filled |
| **Task** | Binary classification: `filled` / `empty` |
| **Export formats** | ONNX (`omr_model.onnx`) |
| **Parameters** | ~120k |

### Architecture

SmallOMRNet uses depthwise separable convolutions (DSConv) for efficiency. Each DSConv block is a depthwise conv + BN + ReLU followed by a pointwise conv + BN + ReLU.

```
Input: (1, 64, 64)
→ Conv2d(1→24, 3×3, stride=2) + BN + ReLU     # (24, 32, 32)
→ DSConv(24→32)                                 # (32, 32, 32)
→ DSConv(32→48, stride=2)                       # (48, 16, 16)
→ DSConv(48→64)                                 # (64, 16, 16)
→ DSConv(64→96, stride=2)                       # (96,  8,  8)
→ DSConv(96→128)                                # (128, 8,  8)
→ AdaptiveAvgPool2d(1) → Flatten                # (128,)
→ Linear(128→64) + ReLU + Dropout(0.20)
→ Linear(64→1)                                  # logit
→ Sigmoid                                       # fill probability ∈ [0, 1]
```

---

## Training

### Dataset

Synthetic checkbox crops generated from PDF answer-sheet templates using the scripts in this repository. Each crop is a padded region around a single checkbox (pad ratio 0.35), rendered at 300 DPI from the template PDF.

- **Classes**: `filled` (label 1) / `empty` (label 0)
- **Image size**: 64 × 64 grayscale
- **Normalisation**: mean=0.5, std=0.5

### Augmentation (training only)

| Transform | Parameters |
|---|---|
| RandomAffine | rotate ±8°, translate ±6%, scale 92–108%, shear ±5° |
| RandomPerspective | distortion 0.18, p=0.25 |
| ColorJitter | brightness 0.15, contrast 0.15 |
| GaussianBlur | kernel 3, σ ∈ [0.1, 1.0] |
| RandomErasing | p=0.10, scale 2–10% |

### Hyperparameters

| Parameter | Value |
|---|---|
| Optimizer | AdamW |
| Learning rate | 0.001 |
| Weight decay | 0.0001 |
| LR schedule | CosineAnnealingLR |
| Loss | BCEWithLogitsLoss (class-balanced pos_weight) |
| Batch size | 128 |
| Max epochs | 20 |
| Early stopping | patience=5, monitor=ROC-AUC |
| Random seed | 42 |

---

## Inference

### ONNX (recommended)

```python
import cv2
import numpy as np
import onnxruntime as ort

IMG_SIZE = 64
MEAN, STD = 0.5, 0.5

session = ort.InferenceSession("omr_model.onnx", providers=["CPUExecutionProvider"])
input_name = session.get_inputs()[0].name

def predict(crop_bgr: np.ndarray) -> float:
    """Return fill probability for a single BGR checkbox crop."""
    gray = cv2.cvtColor(crop_bgr, cv2.COLOR_BGR2GRAY)
    resized = cv2.resize(gray, (IMG_SIZE, IMG_SIZE), interpolation=cv2.INTER_LINEAR)
    arr = (resized.astype(np.float32) / 255.0 - MEAN) / STD
    batch = arr[np.newaxis, np.newaxis, :, :]          # (1, 1, 64, 64)
    logit = session.run(None, {input_name: batch})[0]
    return float(1.0 / (1.0 + np.exp(-logit)))        # sigmoid

crop = cv2.imread("path/to/checkbox_crop.png")
prob = predict(crop)
print("filled" if prob >= 0.5 else "empty", f"  prob={prob:.3f}")
```

### Decision thresholds

| Parameter | Default | Env var override |
|---|---|---|
| Fill threshold | 0.50 | `OMR_THRESHOLD` |
| Uncertain margin | 0.10 | `OMR_UNCERTAIN_MARGIN` |
| Min confidence | 0.55 | `OMR_MIN_CONFIDENCE` |

A prediction is flagged `uncertain` when `|prob - threshold| < uncertain_margin` or `confidence < min_confidence`, where `confidence = 1 - normalised_entropy(prob)`.

---

## Repository Contents

| Path | Description |
|---|---|
| `omr_model.onnx` | Trained model — primary deployment artifact |
| `config/config.yaml` | Full pipeline config: template variants, detector settings, training hyperparameters |
| `data/raw/template/` | Answer-sheet template PDFs (6 variants: 2-/4-/6-choice, mixed, assembled, math quiz) |
| `data/template_map/` | JSON coordinate maps for each template variant |

---

## Template Variants

Six answer-sheet layouts are supported. Each variant has a corresponding template PDF and a pre-built template map (checkbox coordinate file).

| Variant | Choices per question | Template PDF |
|---|---|---|
| `2choice` | A, B | `template_2choice.pdf` |
| `4choice` | A, B, C, D | `template_4choice.pdf` |
| `6choice` | A, B, C, D, E, F | `template_6choice.pdf` |
| `mixed` | variable | `template_mixed.pdf` |
| `assembled_4choice` | A, B, C, D | `template_4choice_assembled.pdf` |
| `math_quiz` | A, B, C, D | `exam_math_quiz.pdf` |

---

## Intended Use

Designed for automated grading pipelines that pre-crop answer-sheet checkboxes (after scan alignment and template matching) and need a fast, CPU-friendly classifier to decide filled vs. empty. Intended to run as part of the [plom-digit-recognition-server](https://gitlab.com/plom/plom).

**Out of scope**: general object detection, handwriting recognition, forms with non-checkbox mark types.