--- license: other license_name: mit-apache-2.0-dinov3 license_link: https://huggingface.co/bxding/RC1/blob/main/NOTICE.md library_name: onnx pipeline_tag: image-classification base_model: - timm/mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k - timm/convnext_tiny.dinov3_lvd1689m datasets: - bxding/recaptcha-67k tags: - computer-vision - image-classification - onnx --- # recaptcha-classifier (RC1) RC1 is a pair of small, fast image recognition models that select the matching cells in reCAPTCHA v2 image grids. The 3×3 model classifies each of the nine cells separately, while the 4×4 model processes the whole grid at once. Both run locally on a CPU with ONNX Runtime. On my $2 VPS, the 3×3 model processes about **9 grids per second** using two CPU threads. ![Example predictions](assets/demo.webp) *Inference in this demo ran on my $2 VPS. The left window shows a 4×4 grid and the right window shows a 3×3 grid.* ## Models | Model | Backbone | Input | Supported targets | | --- | --- | --- | --- | | `grid9.onnx` | MobileNetV4 (`mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k`) | Nine cells from a 3×3 grid, each resized to 224×224 | bicycle, bus, car, crosswalk, fire hydrant, motorcycle\*, parking meter\*, stairs\*, traffic light\* | | `grid16.onnx` | ConvNeXt Tiny (`convnext_tiny.dinov3_lvd1689m`) | One 4×4 grid image, resized to 448×448 | bicycle, bus, chimney\*, crosswalk, fire hydrant, motorcycle, parking meter\*, stairs, taxi\*, tractor, traffic light | `grid16.onnx` is built with DINOv3 and is distributed under the [DINOv3 License](LICENSE-DINOv3.md), not the MIT License. See [License](#license). \* These targets are rare. Across all splits, the 3×3 dataset has 14 grids for motorcycle, 7 for traffic light, 2 for parking meter, and 1 for stairs. The 4×4 dataset has 7 grids for parking meter, 13 for chimney, and 108 for taxi. Each model outputs a probability for every cell. The inference script selects the cells at or above one tuned threshold per model: 0.475 for `grid9.onnx` and 0.45 for `grid16.onnx`. The threshold is stored in the model file. ## Evaluation Accuracy of the released ONNX models on [recaptcha-67k](https://huggingface.co/datasets/bxding/recaptcha-67k). *Exact* means every cell prediction matched the reference label. | Model | Split | Grids | Exact | Cell precision | Cell recall | Cell F1 | | --- | --- | ---: | ---: | ---: | ---: | ---: | | `grid9.onnx` | `review` | 300 | 81.0% | 0.978 | 0.948 | 0.963 | | `grid9.onnx` | `test` (random 600) | 600 | 82.5% | 0.976 | 0.958 | 0.967 | | `grid16.onnx` | `review` | 282 | 63.5% | 0.915 | 0.939 | 0.927 | | `grid16.onnx` | `test` (random 600) | 600 | 65.7% | 0.944 | 0.948 | 0.946 | These numbers measure agreement with model-generated labels, not live challenge pass rates: - The `test` labels come from the same labelers RC1 was trained on: Qwen for 3×3 and SAM 3 for 4×4. - The `review` labels were checked by Claude Opus 5.5, not by humans. - The original machine labels exactly matched Claude’s review on 81.3% of 3×3 grids and 70.6% of 4×4 grids. Among 4×4 targets with at least 10 reviewed grids, exact agreement was lowest for stairs (10/18), traffic lights (29/48), and motorcycles (52/86). ## Running inference Requires Python 3.11 or newer. From this directory, install the dependencies and run the prediction script: ```sh python -m pip install onnxruntime numpy pillow python predict_onnx.py --model weights/grid16.onnx --image challenge.jpg --target buses ``` Use `weights/grid9.onnx` for 3×3 grids and `weights/grid16.onnx` for 4×4 grids. The script prints JSON with a probability for each cell and a `selected_cells` list. Cells are numbered from 0, left to right and then top to bottom. To limit the number of CPU threads, pass `--threads`. For all options, run `python predict_onnx.py --help`. ## Benchmarks Approximate CPU throughput once the model is loaded. Times include image loading and preprocessing. | Machine | CPU threads | 3×3 grids/sec | 4×4 grids/sec | | --- | ---: | ---: | ---: | | Apple M2 Pro | 4 | ~10 | ~7 | | $2 VPS | 2 | ~9 | ~4 | ## Training and export Training requires Python 3.11, [uv](https://docs.astral.sh/uv/), and a local copy of the recaptcha-67k dataset. From this directory: ```sh # Install dependencies uv sync --locked # Prepare the dataset uv run python train.py prepare --data /path/to/recaptcha-67k --output runs/rc # Train the 3×3 and 4×4 models uv run python train.py train --manifest runs/rc/split.json --output runs/rc --tiles 9 uv run python train.py train --manifest runs/rc/split.json --output runs/rc --tiles 16 --backbone convnext_tiny.dinov3_lvd1689m # Export to ONNX uv run --group export python train.py export --checkpoint runs/rc/grid9/best.pt # Run tests uv run --group export python -m unittest test_training ``` Notes: - Training uses a GPU if one is available. On CUDA, add `--amp` to reduce memory usage. - The `prepare` step keeps the dataset's existing splits and leaves out the `excluded` split. - All training outputs are written locally to `runs/`. - For dataset and training options, run `python train.py --help` or add `--help` to any subcommand. ## Roadmap - Improve 4×4 accuracy, which trails both the 3×3 model and the SAM 3 labels it was trained on. - Improve chimney and parking meter recognition with more training data. - Add training data and model support for bridges. ## Disclaimer RC1 was created for educational and research purposes. It is not affiliated with or endorsed by Google or reCAPTCHA. Users are responsible for complying with all applicable laws and terms of service. ## License RC1's files are under three licenses. [NOTICE.md](NOTICE.md) has the details. | Files | License | | --- | --- | | Source code and documentation | [MIT](LICENSE) | | `weights/grid9.onnx` | [MIT](LICENSE), plus [Apache-2.0](LICENSE-APACHE-2.0.txt) for its pretrained [MobileNetV4 backbone](https://huggingface.co/timm/mobilenetv4_conv_medium.e250_r384_in12k_ft_in1k) | | `weights/grid16.onnx` | [DINOv3 License](LICENSE-DINOv3.md) only | `grid16.onnx` is fine-tuned from Meta's [DINOv3 ConvNeXt Tiny](https://huggingface.co/timm/convnext_tiny.dinov3_lvd1689m) (built with DINOv3). The DINOv3 License requires derivatives to be distributed under its own terms, so the MIT License does not apply to this file. Anyone who uses or redistributes it must follow the DINOv3 License, including its trade control and use restrictions. Third-party imagery in the training dataset and demo is outside the scope of these licenses.