Cianmcnally/idm-v10-checkpoints

Click IDM — Cursor Streams

Given only screen pixels, decide on which frame a physical mouse button went down.

A video shows you what happened on a screen, but not what the person did. A click is invisible unless you can read the 2–3 frame press animation under the cursor. This model reads it. Solve that, and every unlabelled screen recording becomes action-labelled training data.

Trained on Cursor Streams — 156 hours of human desktop use with frame-accurate click labels and dense per-frame cursor boxes.

Click position is an input, never a prediction. The cursor track is given. The model only has to answer when.

Checkpoints

Two checkpoints won two different objectives, and both are here.

file	role	validation AP
best_val_ap_and_ood_f1__c_v16_width640_drop05_ft__epoch01.pt	validation-AP specialist	0.855813123
c_v18_seed101__epoch02.pt	OOD ensemble component, weight 0.25	0.833632303
c_v18_seed202__epoch02.pt	OOD ensemble component, weight 0.75	0.832994669

If you want one model, take the v16 file — it holds the best validation average precision measured on this project. If you care about out-of-distribution robustness, the deployed configuration is the v18 pair blended 0.25 / 0.75, which was selected on OOD F1 rather than AP. Neither v18 file reproduces the deployment number alone.

Architecture

Dual-stream CNN encoder over a temporal trunk, in the lineage described in the dataset card (dual-stream CNN + non-causal dilated TCN). Verified from the checkpoint itself:

Parameters	16,522,831 (16.52 M)
Tensors	136
Trunk width	640
Dropout	0.05
Mode	fused
Frame differencing	enabled (use_diff: true)
Metadata stream	enabled (use_metadata: true)
ROI stream	disabled (use_roi: false)
Sub-frame offset range	±4.0 frames
File size	~189 MB (full training state, not weights alone)

The width-640 trunk is the sweet spot, and that is measured rather than assumed — see Width ablation below.

Inputs

The model consumes a 64-frame clip sampled at stride 5 — every 5th frame of 30 fps source video, so ~6 fps covering ~11 seconds of context. Sampling at 6 fps rather than 30 is load-bearing: identical models scored roughly 5× higher F1 at stride 5 than stride 1. A 30 fps clip is mostly redundant frames and too little context.

Per frame, two views plus a metadata vector:

stream	shape	meaning
local	64 × 64	cursor-anchored crop, taken from a 600 px field of view
global	192 × 120	downscaled window/screen view
metadata	vector	cursor kinematics and context (normalised by metadata_stats stored in the checkpoint)

The local crop must be anchored on the cursor, not the frame centre. An earlier bug centred it on the frame; whenever the crop clamped at a screen edge the cursor left the input entirely — on ~30 % of click frames in some sessions. Every number produced before that fix was invalid.

Up to 8 events are handled per clip (max_events: 8).

Outputs

Per-frame logits plus a sub-frame regression. The training loss supervises:

term	weight	meaning
down	1.0	mouse button pressed on this frame
up	1.0	button released
click	0.5	click-level supervision
held	0.5	button currently held
moving	0.2	cursor in motion
offset	0.25	sub-frame timing residual, range ±4 frames
consistency	0.1	regulariser
negpeak	0.30	near-click false-positive suppression (negpeak_k: 8)

The offset head matters: down_frame labels in Cursor Streams are fractional on purpose — the press happened between two frames. The model predicts that residual rather than rounding it away.

Decoding

Raw per-frame probabilities need peak-picking. The calibrated decoder for the v16 checkpoint, grid-searched over 300 points on validation only (72 sessions, zero failures):

{ "down_thresh": 0.55, "up_thresh": 0.55, "refractory": 3,
  "prominence": 0.0, "temperature": 1.0 }

For the v18 ensemble, blend probabilities across components first (probability_blend: true, offset_blend: "probability_weighted"), then decode with refractory: 1 instead of 3.

Results

Validation, pooled across val, val_psai, val_a11ycua, val_videocua (72 sessions), using the v16 checkpoint at its calibrated decoder:

metric	value
Average precision	0.855813123
F1	0.6829
Precision	0.7165
Recall	0.6523
TP / FP / FN	364 / 144 / 194

Individual v18 seeds scored 0.8336 (seed101), 0.8330 (seed202), 0.8347 (seed303) — all below v16 on validation AP, which is exactly why the v16 checkpoint is the AP specialist and the v18 pair is kept for its OOD behaviour instead.

These are validation numbers. Do not read them as held-out test performance.

Training

Fine-tuning stage (ft) — low learning rate, three epochs, from an earlier trunk.

model: {mode: fused, use_diff: true, use_metadata: true, use_roi: false,
        offset_range: 4.0, trunk_width: 640, dropout: 0.05}

data:  {clip_len: 64, frame_stride: 5, global_h: 120, global_w: 192,
        local_fov_px: 600, local_size: 64, max_events: 8,
        samples_per_epoch: 6000,
        mixture: {click: 0.30, drag: 0.10, hardneg: 0.40, bg: 0.20},
        bg_span_clear: true,
        soft_targets: true, soft_sigma: 0.7, soft_radius: 1.4,
        soft_ring: 2.4, offset_radius: 3.5,
        source_mixture: {recorder: 0.10, a11y-cua: 0.25,
                         psai-clicks: 0.45, videocua: 0.20}}

loss:  {click_max_frames: 15, focal_alpha: 0.25, focal_gamma: 0.0,
        offset_enabled: true, target_mass: 1.5,
        w_down: 1.0, w_up: 1.0, w_click: 0.5, w_held: 0.5, w_moving: 0.2,
        w_offset: 0.25, w_consistency: 0.1, w_negpeak: 0.30, negpeak_k: 8}

train: {batch_size: 16, epochs: 3, lr: 0.00002, weight_decay: 0.0001,
        grad_accum: 1, grad_clip: 1.0, warmup_frac: 0.05, num_workers: 4,
        save_every_epoch: true, early_stop_patience: 99,
        val_peak_thresh: 0.5, val_refractory: 4, val_tolerance: 1.4}

Note the 40 % hard negatives in the sample mixture — the dominant failure mode is near-click double-firing, not hallucination on quiet screens, so the mixture is weighted against it.

Soft targets are the other load-bearing choice

soft_targets: true, soft_sigma: 0.7 with a don't-care ring at 2.4. A one-frame hard target evaluated at a ±7-frame tolerance trains the very frames you will later score as correct as hard background; the model resolves that contradiction with a smeared, multi-peak response. Matching the target to the evaluation ruler cut near-click false positives 40–50 % and background false positives 38–63 % in a single change.

Width ablation

A later sweep (v19) at four capacities, shared-batch, epoch 3 — best validation AP per arm:

trunk width	dropout	val AP
640	0.02	0.839
640	0.10	0.838
704	—	0.818
768	—	0.810

Widening the trunk monotonically hurts, and it hurts more the wider it goes. 640 is the right size. Dropout between 0.02 and 0.10 is a coin flip at this separation. None of these arms beat the 0.8558 held by the width-640 / dropout-0.05 fine-tune above.

Usage

Each file is a full training checkpoint, not a bare state dict — it carries optimizer, scheduler, RNG states and metrics alongside the weights. Pull the weights out of the model key:

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    "Cianmcnally/idm-v10-checkpoints",
    "best_val_ap_and_ood_f1__c_v16_width640_drop05_ft__epoch01.pt",
)
ckpt = torch.load(path, map_location="cpu", weights_only=False)

state = ckpt["model"]              # 136 tensors, 16.52 M params
cfg   = ckpt["cfg"]                # architecture config as trained
stats = ckpt["metadata_stats"]     # normalisation for the metadata stream — you need this
print(ckpt["epoch"], ckpt["val_ap"], ckpt["val_f1"])

Available keys: model, optimizer, scheduler, global_step, cfg, epoch, metadata_stats, val_f1, val_ap, best_ap, best_fp, bad_epochs, torch_rng, cuda_rng, numpy_rng.

Resuming training is therefore possible directly from these files — the optimizer moments are why each is ~189 MB rather than ~66 MB.

Ensembling the v18 pair:

p = 0.25 * p_seed101 + 0.75 * p_seed202     # blend probabilities, not logits
# then decode with down/up thresh 0.55, refractory 1

Limitations

• No macOS. Training data is Windows and Linux desktops only.
• Reported metrics are validation, not test. Treat them as a model-selection signal.
• The metadata normalisation is not optional — feed the model unnormalised metadata and results will not reproduce. Use the metadata_stats in the checkpoint.
• Timing, not detection, is the remaining error. At a ±7 source-frame ruler (±233 ms) the model does well; at ±2 frames scores drop substantially.
• The model architecture definition is not included here — these are weights and config. You need the idm_v10 training code to instantiate the module.
• No inference wrapper or ONNX export is provided.

Attribution

Trained on Cursor Streams, which is CC-BY-4.0 and itself derives from ServiceNow/VideoCUA (MIT), computer-use-data-psai (MIT), and berkeley-hci/A11y-CUA (CC-BY-4.0). The attribution requirement carries through — if you use this model, cite those corpora too.

@misc{mcnally2026clickidm,
  title  = {Click IDM: frame-accurate mouse-press recovery from screen video},
  author = {McNally, Cian},
  year   = {2026},
  url    = {https://huggingface.co/Cianmcnally/idm-v10-checkpoints}
}