--- license: apache-2.0 language: - en base_model: - microsoft/Florence-2-base-ft datasets: - inclusionAI/VenusBench-CAPTCHA - OpenCaptchaWorld/Open_CaptchaWorld - hammer888/captcha-data pipeline_tag: image-text-to-text library_name: onnxruntime model-index: - name: fiptcha results: - task: type: image-text-to-text name: CAPTCHA Visual Grounding dataset: name: VenusBench-CAPTCHA (val.jsonl, 39 grounding samples) type: inclusionAI/VenusBench-CAPTCHA split: validation metrics: - type: accuracy value: 79.5 name: IoU@0.5 - type: iou value: 0.669 name: Mean IoU tags: - onnx - int8 - florence2 - captcha - vision-language - browser-automation - computer-use ---

Fiptcha

npm  •  GitHub  •  Apache 2.0 License

**Fiptcha** is a lightweight, CPU-efficient vision-language CAPTCHA solver built for browser automation within tight memory constraints. Fine-tuned from [`microsoft/Florence-2-base-ft`](https://huggingface.co/microsoft/Florence-2-base-ft) on CAPTCHA challenge data derived from [`inclusionAI/VenusBench-CAPTCHA`](https://huggingface.co/datasets/inclusionAI/VenusBench-CAPTCHA) and dynamically quantized to **INT8 ONNX (~431 MB total across 5 submodels)**, Fiptcha runs purely via `onnxruntime` and `transformers` / `tokenizers`—requiring no GPU or PyTorch at inference time. Fiptcha is designed to act as a local CAPTCHA solving engine (`` and `` for text, icon/text click, slider puzzle, drag-end, image rotation, and visual reasoning challenges) on resource-constrained laptops, Apple Silicon devices, and CPU-only containers. | Artifact | Backbone | Precision / Format | Disk Size | | :--- | :--- | :--- | :--- | | **ONNX INT8 Submodels (Default)** | `microsoft/Florence-2-base-ft` | Dynamic per-channel `QInt8` ONNX | ~431 MB (total) | | **`fiptcha-florence2-merged` (Checkpoint)** | `microsoft/Florence-2-base-ft` | `float32` Safetensors | ~883.4 MB | ### Submodel Breakdown | File | INT8 Size | Role | | :--- | :---: | :--- | | `vision_encoder.onnx` | 90 MB | DaViT vision backbone + 2D/temporal embeddings + projection | | `embed_tokens.onnx` | 38 MB | Shared token embedding lookup | | `encoder_model.onnx` | 42 MB | Multimodal BART-style 6-layer encoder | | `decoder_model.onnx` | 131 MB | Autoregressive 6-layer decoder (initial step) | | `decoder_with_past_model.onnx` | 124 MB | Autoregressive 6-layer decoder with KV cache | ### Inputs and outputs * **Input:** * A CAPTCHA screenshot / image resized and normalized to `768x768` (`CLIPImageProcessor`). * A Florence-2 task prompt: * `` for distorted-text / OCR CAPTCHAs * `` (e.g., `slider puzzle target`, `icon click target`, `text click target`, `drag end target`, `image rotation target`, `visual reasoning target`) * **Output:** * Structured action or grounding output, such as `Click(box=(x, y)),Type(content='...')` for `` or `` bounding-box coordinates (0–999 scale) for ``. ### Citation ```bibtex @misc{fiptcha_2026, title = {Fiptcha: Lightweight CPU-Efficient Local CAPTCHA Solver}, author = {Figranium}, year = {2026}, url = {https://huggingface.co/figranium/fiptcha} } ``` --- ## Model Data Data used for model training and how the data was processed. ### Training Dataset Fiptcha was fine-tuned on a combined mixture of three public CAPTCHA datasets (augmented with repetition factors and split into `train.jsonl` and `val.jsonl`), plus an optional local click-target set: * **[`inclusionAI/VenusBench-CAPTCHA`](https://huggingface.co/datasets/inclusionAI/VenusBench-CAPTCHA)** (`test` split, upsampled `15x`): * **OCR CAPTCHAs:** Text-based challenges parsed from `action_raw` and annotated with `` prompts. * **Visual / Grounding CAPTCHAs:** Click and interaction target boxes annotated with `{prompt}` to predict localized target boxes (``). * **[`OpenCaptchaWorld/Open_CaptchaWorld`](https://huggingface.co/datasets/OpenCaptchaWorld/Open_CaptchaWorld)** (`train` split, upsampled `10x`): * **Point-Click / Multi-Point CAPTCHAs:** Point coordinates `(x, y)` mapped into padded bounding boxes for visual localization prompts. * **VQA / Question-Answering CAPTCHAs:** General alphanumeric text and label verification questions mapped to a `` prefix and target answers. * **[`hammer888/captcha-data`](https://huggingface.co/datasets/hammer888/captcha-data)** (`40,000` samples): * Distorted text-based images representing classic visual OCR CAPTCHAs, mapped to the `` task prefix. ### Data Preprocessing * **Image Preprocessing:** Collected images were cached and normalized into `train.jsonl` / `val.jsonl` catalogs, resized to `768x768` (`resample=3`), and normalized with standard ImageNet/CLIP mean (`[0.485, 0.456, 0.406]`) and std (`[0.229, 0.224, 0.225]`), producing `577` visual tokens. * **Prompt Formatting:** Each example is formatted as `(image, prefix, suffix)` pairs across ``, ``, and `` task prefixes. --- ## Implementation Information Details about the model internals and runtime optimization. * **Architecture:** `Florence2ForConditionalGeneration` — DaViT vision encoder (`depths=[1, 1, 9, 1]`, `dim_embed=[128, 256, 512, 1024]`) paired with a `6`-layer encoder / `6`-layer decoder BART-style language model (`d_model=768`, `12` attention heads, `vocab_size=51,289`). * **Quantization & Export:** Exported from PyTorch (`attn_implementation="eager"`) to FP32 ONNX (`opset_version=17`) with dynamic batch and sequence axes, then quantized using `onnxruntime.quantization.quantize_dynamic` (`per_channel=True`, `weight_type=QuantType.QInt8`, `MatMulConstBOnly=True`). --- ## Evaluation & Benchmarks Fiptcha (`ONNX INT8`) is evaluated on the held-out `VenusBench-CAPTCHA` validation split (`val.jsonl`, **44 samples** across 6 visual grounding categories and OCR). ### `VenusBench-CAPTCHA` Validation Results (INT8 ONNX · CPU) | Category | Samples | IoU@0.5 | Mean IoU | | :--- | :---: | :---: | :---: | | **`drag end target`** | 8 | **100.0%** (8/8) | 0.903 | | **`image rotation target`** | 3 | **100.0%** (3/3) | 0.940 | | **`text click target`** | 8 | **87.5%** (7/8) | 0.641 | | **`slider puzzle target`** | 6 | **83.3%** (5/6) | 0.715 | | **`visual reasoning target`** | 6 | **83.3%** (5/6) | 0.644 | | **`icon click target`** | 8 | **37.5%** (3/8) | 0.346 | | **Grounding Overall** | **39** | **79.5%** (31/39) | **0.669** | - **OCR (`5` samples):** 40.0% (2/5) exact match on `Type(content='...')`. - **Latency (CPU, greedy `num_beams=1`):** **2.523s** median (**2.526s** mean) per sample using `onnxruntime`. ### Methodology & Setup - **Task Formulation:** Each sample is evaluated with its corresponding `` or `` prompt using greedy decoding (`num_beams=1`, `max_new_tokens=64`) and `skip_special_tokens=False` to preserve `` coordinate tokens. - **Grounding Metric:** For each ground-truth bounding box, we take the maximum IoU against all predicted bounding boxes, average across ground-truth boxes in the sample, and report `IoU@0.5` and `Mean IoU`. --- ## Ethics and Safety Fiptcha is intended for authorized browser automation testing, accessibility workflows, and research on CAPTCHA robustness. Users are responsible for complying with applicable terms of service and laws when automating interactions with web applications. --- ## Usage and Limitations ### Usage Install the Node.js runtime from [`github.com/figranium/fiptcha`](https://github.com/figranium/fiptcha): ```bash npm install fiptcha ``` To download the quantized ONNX INT8 model weights locally from Hugging Face: ```bash hf download figranium/fiptcha --local-dir ./fiptcha-int8 ``` ### System Requirements * **Disk Space:** ~431 MB (`vision_encoder.onnx`, `embed_tokens.onnx`, `encoder_model.onnx`, `decoder_model.onnx`, `decoder_with_past_model.onnx`, plus tokenizer/processor files) * **Hardware:** Standard x86_64 or ARM64 CPU; no GPU required ### Limitations * **Domain Specificity:** Fine-tuned primarily on UI/web CAPTCHA screenshots (`VenusBench-CAPTCHA`); performance degrades on out-of-domain low-resolution synthetic text CAPTCHAs. * **Single-Step Inference:** Produces a single `` or `` prediction per image; multi-round challenge orchestration is handled by the `fiptcha` Node.js runtime.