| --- |
| model_name: Yap Phone Screenshot Classifier |
| library_name: 'timm' |
| base_model: 'timm/mobilenetv4_conv_medium.e250_r384_in12k' |
| tags: |
| - image-classification |
| - mobile-screenshots |
| - phone-screenshots |
| - screenshot-analysis |
| - content-safety |
| - timm |
| datasets: |
| - yapwithai/phone-screenshots |
| --- |
| |
| # Yap Phone Screenshot Classifier |
|
|
| Yap Phone Screenshot Classifier predicts two labels for one image: |
|
|
| 1. `screen`: the phone-screen category. |
| 2. `safety`: the content-safety category. |
|
|
| The model returns exactly two outputs: `screen` and `safety`. |
|
|
| The model is fine-tuned from `timm/mobilenetv4_conv_medium.e250_r384_in12k`. |
|
|
| ## What It Is For |
|
|
| This model is built for routing and filtering mobile screenshot workflows. It separates UI classification from safety classification so an application can answer both of these questions before running more expensive analysis: |
|
|
| - What kind of screen is this? |
| - Which safety category does this image belong to? |
|
|
| `generic` and `other` are real screen labels when included in the exported label map. `generic` means phone UI that is not one of the more specific trained screen classes. `other` means non-phone UI or images outside the phone-screenshot distribution. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| | :------------------------ | :------------------------------------------------------------- | |
| | `onnx/model.onnx` | ONNX model for CPU/server inference. | |
| | `onnx/model.fp16.onnx` | Optional FP16 ONNX candidate. | |
| | `onnx/model.onnx.data` | External ONNX weight data loaded beside `model.onnx`. | |
| | `model.safetensors` | PyTorch state dict for reproducibility and continued training. | |
| | `config.json` | Model identity, base model, output names, and label arrays. | |
| | `preprocess.json` | Resize and normalization contract used during training/export. | |
| | `train.json` | Sanitized training recipe for post-training or reproduction. | |
| | `inference/python.py` | Python helper for ONNX inference from image paths. | |
| | `inference/typescript.ts` | TypeScript helper for ONNX inference from image paths. | |
| | `inference/labels.json` | Screen and safety labels used to decode model logits. | |
| | `README.md` | This model card and runnable inference reference. | |
|
|
| ## Outputs |
|
|
| The ONNX graph has two outputs in this exact order: |
|
|
| | Output | Shape | Meaning | |
| | :------- | :---------------------------- | :---------------------- | |
| | `screen` | `[batch, screen_class_count]` | Screen-category logits. | |
| | `safety` | `[batch, safety_class_count]` | Content-safety logits. | |
|
|
| Inference uses `argmax` for both heads in this version. |
|
|
| ## Test Results |
|
|
| | Model | Test | Output | Accuracy | Balanced Acc | Macro F1 | Top-2 | Images | |
| | :-- | --: | --: | --: | --: | --: | --: | --: | |
| | fp32 | full test | screen | 0.9154 | 0.8385 | 0.7265 | 0.9719 | 23615 | |
| | fp32 | full test | safety | 0.9575 | 0.8941 | 0.8600 | 0.9921 | 23615 | |
| | fp32 | screen-balanced test | screen | 0.8377 | 0.8421 | 0.8133 | 0.9400 | 6246 | |
| | fp32 | screen-balanced test | safety | 0.9776 | 0.9285 | 0.9110 | 0.9978 | 6246 | |
| | fp32 | safety-balanced test | screen | 0.9580 | 0.7752 | 0.6547 | 0.9870 | 3000 | |
| | fp32 | safety-balanced test | safety | 0.8957 | 0.8957 | 0.8947 | 0.9847 | 3000 | |
| | fp16 | full test | screen | 0.9153 | 0.8384 | 0.7264 | 0.9719 | 23615 | |
| | fp16 | full test | safety | 0.9575 | 0.8941 | 0.8600 | 0.9921 | 23615 | |
| | fp16 | screen-balanced test | screen | 0.8373 | 0.8419 | 0.8130 | 0.9400 | 6246 | |
| | fp16 | screen-balanced test | safety | 0.9776 | 0.9285 | 0.9110 | 0.9978 | 6246 | |
| | fp16 | safety-balanced test | screen | 0.9580 | 0.7752 | 0.6547 | 0.9870 | 3000 | |
| | fp16 | safety-balanced test | safety | 0.8957 | 0.8957 | 0.8947 | 0.9847 | 3000 | |
|
|
| ## CPU Timing |
|
|
| | Model | Images/s | Load Mean | Model Mean | Total Mean | Total Median | Total P95 | Provider | |
| | :-- | --: | --: | --: | --: | --: | --: | --: | |
| | fp32 | 20.0538 | 0.12 ms | 49.52 ms | 49.87 ms | 57.84 ms | 85.82 ms | onnxruntime:CPUExecutionProvider | |
| | fp16 | 20.8351 | 0.11 ms | 47.69 ms | 48.00 ms | 48.60 ms | 63.65 ms | onnxruntime:CPUExecutionProvider | |
|
|
| Timing is measured with ONNX Runtime CPU execution on `Apple M4 Max (16 logical cores)`. Total latency includes image load/preprocess, model inference, and label decoding. |
|
|
| ## Classification Labels |
|
|
| For the screen and safety label lists, see [`config.json`](config.json). |
|
|
| ## Preprocessing |
|
|
| Use the preprocessing contract in `preprocess.json`. |
|
|
| 1. Read the image. |
| 2. Apply EXIF orientation. |
| 3. Convert to RGB. |
| 4. Resize so the longest side is `1024` pixels and keep the original aspect ratio. |
| 5. Do not crop, stretch, square-pad before preprocessing, or horizontally flip. |
| 6. Convert to `float32` in `[0, 1]`. |
| 7. Normalize with the mean and standard deviation from `preprocess.json`. |
| 8. Pad each normalized tensor to the batch maximum height and width, rounded up to a multiple of 32. |
|
|
| The exported model accepts dynamic batch, height, and width. |
|
|
| ## ONNX Inference |
|
|
| The helpers intentionally return only `screen` and `safety`. |
|
|
| Keep `onnx/model.onnx.data` beside `onnx/model.onnx`; ONNX Runtime loads the external tensor data when it opens the graph. |
| By default the helpers load the FP32 model at `onnx/model.onnx`. Pass `fp16` to load `onnx/model.fp16.onnx`. |
|
|
| ### Download |
|
|
| Download the exported model folder from Hugging Face: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| |
| model_dir = snapshot_download("yapwithai/phone-screen-classifier") |
| ``` |
|
|
| ### Python |
|
|
| Install the runtime dependencies: |
|
|
| ```bash |
| python -m pip install numpy pillow onnxruntime |
| ``` |
|
|
| Import the helper from the exported model folder: |
|
|
| ```python |
| from inference.python import Classifier, classify |
| |
| print(classify("example.png")) |
| print(classify("example.png", model_format="fp16")) |
| |
| classifier = Classifier(model_dir) |
| print(classifier.classify_batch(["one.png", "two.png"])) |
| |
| fp16_classifier = Classifier(model_dir, model_format="fp16") |
| print(fp16_classifier.classify_batch(["one.png", "two.png"])) |
| ``` |
|
|
| Or run it directly: |
|
|
| ```bash |
| python inference/python.py example.png another.png --model-format fp16 |
| ``` |
|
|
| ### TypeScript |
|
|
| Install the runtime dependencies: |
|
|
| ```bash |
| bun add sharp onnxruntime-node |
| ``` |
|
|
| Import the helper from the exported model folder: |
|
|
| ```ts |
| import { Classifier, classify } from './inference/typescript.ts'; |
| |
| console.log(await classify('example.png')); |
| console.log(await classify('example.png', modelDir, 'fp16')); |
| |
| const classifier = await Classifier.create(modelDir); |
| console.log(await classifier.classifyBatch(['one.png', 'two.png'])); |
| |
| const fp16Classifier = await Classifier.create(modelDir, 'fp16'); |
| console.log(await fp16Classifier.classifyBatch(['one.png', 'two.png'])); |
| ``` |
|
|
| Or run it directly: |
|
|
| ```bash |
| bun inference/typescript.ts example.png another.png --model-format fp16 |
| ``` |
|
|
| ## Citation |
|
|
| If you use this model, please cite: |
|
|
| ```bibtex |
| @misc{phone-screen-classifier, |
| title={Yap Phone Screenshot Classifier}, |
| year={2026}, |
| publisher={Yap With AI}, |
| url={https://huggingface.co/yapwithai/phone-screen-classifier} |
| } |
| ``` |
|
|
