File size: 6,591 Bytes
f6d09c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 | ---
license: other
license_name: mit-apache-2.0-dinov3
license_link: https://huggingface.co/bxding/RC1/blob/main/NOTICE.md
library_name: onnx
pipeline_tag: image-classification
base_model:
- timm/mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k
- timm/convnext_tiny.dinov3_lvd1689m
datasets:
- bxding/recaptcha-67k
tags:
- computer-vision
- image-classification
- onnx
---
# recaptcha-classifier (RC1)
RC1 is a pair of small, fast image recognition models that select the matching
cells in reCAPTCHA v2 image grids. The 3×3 model classifies each of the nine
cells separately, while the 4×4 model processes the whole grid at once. Both
run locally on a CPU with ONNX Runtime. On my $2 VPS, the 3×3 model processes
about **9 grids per second** using two CPU threads.

*Inference in this demo ran on my $2 VPS. The left window shows a 4×4 grid and
the right window shows a 3×3 grid.*
## Models
| Model | Backbone | Input | Supported targets |
| --- | --- | --- | --- |
| `grid9.onnx` | MobileNetV4 (`mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k`) | Nine cells from a 3×3 grid, each resized to 224×224 | bicycle, bus, car, crosswalk, fire hydrant, motorcycle\*, parking meter\*, stairs\*, traffic light\* |
| `grid16.onnx` | ConvNeXt Tiny (`convnext_tiny.dinov3_lvd1689m`) | One 4×4 grid image, resized to 448×448 | bicycle, bus, chimney\*, crosswalk, fire hydrant, motorcycle, parking meter\*, stairs, taxi\*, tractor, traffic light |
`grid16.onnx` is built with DINOv3 and is distributed under the
[DINOv3 License](LICENSE-DINOv3.md), not the MIT License. See [License](#license).
\* These targets are rare. Across all splits, the 3×3 dataset has 14 grids
for motorcycle, 7 for traffic light, 2 for parking meter, and 1 for stairs.
The 4×4 dataset has 7 grids for parking meter, 13 for chimney, and 108 for taxi.
Each model outputs a probability for every cell. The inference script selects
the cells at or above one tuned threshold per model: 0.475 for `grid9.onnx`
and 0.45 for `grid16.onnx`. The threshold is stored in the model file.
## Evaluation
Accuracy of the released ONNX models on
[recaptcha-67k](https://huggingface.co/datasets/bxding/recaptcha-67k).
*Exact* means every cell prediction matched the reference label.
| Model | Split | Grids | Exact | Cell precision | Cell recall | Cell F1 |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| `grid9.onnx` | `review` | 300 | 81.0% | 0.978 | 0.948 | 0.963 |
| `grid9.onnx` | `test` (random 600) | 600 | 82.5% | 0.976 | 0.958 | 0.967 |
| `grid16.onnx` | `review` | 282 | 63.5% | 0.915 | 0.939 | 0.927 |
| `grid16.onnx` | `test` (random 600) | 600 | 65.7% | 0.944 | 0.948 | 0.946 |
These numbers measure agreement with model-generated labels, not live
challenge pass rates:
- The `test` labels come from the same labelers RC1 was trained on: Qwen for
3×3 and SAM 3 for 4×4.
- The `review` labels were checked by Claude Opus 5.5, not by humans.
- The original machine labels exactly matched Claude’s review on 81.3% of
3×3 grids and 70.6% of 4×4 grids.
Among 4×4 targets with at least 10 reviewed grids, exact agreement was lowest
for stairs (10/18), traffic lights (29/48), and motorcycles (52/86).
## Running inference
Requires Python 3.11 or newer. From this directory, install the dependencies
and run the prediction script:
```sh
python -m pip install onnxruntime numpy pillow
python predict_onnx.py --model weights/grid16.onnx --image challenge.jpg --target buses
```
Use `weights/grid9.onnx` for 3×3 grids and `weights/grid16.onnx` for 4×4 grids.
The script prints JSON with a probability for each cell and a `selected_cells`
list. Cells are numbered from 0, left to right and then top to bottom.
To limit the number of CPU threads, pass `--threads`. For all options, run
`python predict_onnx.py --help`.
## Benchmarks
Approximate CPU throughput once the model is loaded. Times include image
loading and preprocessing.
| Machine | CPU threads | 3×3 grids/sec | 4×4 grids/sec |
| --- | ---: | ---: | ---: |
| Apple M2 Pro | 4 | ~10 | ~7 |
| $2 VPS | 2 | ~9 | ~4 |
## Training and export
Training requires Python 3.11, [uv](https://docs.astral.sh/uv/), and a local
copy of the recaptcha-67k dataset. From this directory:
```sh
# Install dependencies
uv sync --locked
# Prepare the dataset
uv run python train.py prepare --data /path/to/recaptcha-67k --output runs/rc
# Train the 3×3 and 4×4 models
uv run python train.py train --manifest runs/rc/split.json --output runs/rc --tiles 9
uv run python train.py train --manifest runs/rc/split.json --output runs/rc --tiles 16 --backbone convnext_tiny.dinov3_lvd1689m
# Export to ONNX
uv run --group export python train.py export --checkpoint runs/rc/grid9/best.pt
# Run tests
uv run --group export python -m unittest test_training
```
Notes:
- Training uses a GPU if one is available. On CUDA, add `--amp` to reduce
memory usage.
- The `prepare` step keeps the dataset's existing splits and leaves out the
`excluded` split.
- All training outputs are written locally to `runs/`.
- For dataset and training options, run `python train.py --help` or add
`--help` to any subcommand.
## Roadmap
- Improve 4×4 accuracy, which trails both the 3×3 model and the SAM 3 labels
it was trained on.
- Improve chimney and parking meter recognition with more training data.
- Add training data and model support for bridges.
## Disclaimer
RC1 was created for educational and research purposes. It is not affiliated
with or endorsed by Google or reCAPTCHA. Users are responsible for complying
with all applicable laws and terms of service.
## License
RC1's files are under three licenses. [NOTICE.md](NOTICE.md) has the details.
| Files | License |
| --- | --- |
| Source code and documentation | [MIT](LICENSE) |
| `weights/grid9.onnx` | [MIT](LICENSE), plus [Apache-2.0](LICENSE-APACHE-2.0.txt) for its pretrained [MobileNetV4 backbone](https://huggingface.co/timm/mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k) |
| `weights/grid16.onnx` | [DINOv3 License](LICENSE-DINOv3.md) only |
`grid16.onnx` is fine-tuned from Meta's
[DINOv3 ConvNeXt Tiny](https://huggingface.co/timm/convnext_tiny.dinov3_lvd1689m)
(built with DINOv3). The DINOv3 License requires derivatives to be distributed
under its own terms, so the MIT License does not apply to this file. Anyone who
uses or redistributes it must follow the DINOv3 License, including its trade
control and use restrictions.
Third-party imagery in the training dataset and demo is outside the scope of
these licenses.
|