File size: 5,089 Bytes
5124c2a 21298ad 5124c2a 21298ad 56ae190 21298ad 0355389 21298ad 0355389 f76be3b 0355389 c96df97 0355389 56ae190 0355389 56ae190 0355389 56ae190 0355389 56ae190 0355389 56ae190 0355389 56ae190 0355389 56ae190 0355389 56ae190 0355389 02be520 21298ad | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | ---
license: apache-2.0
datasets:
- lyte-codes/clockface-synth-dataset-10k
pipeline_tag: image-to-text
tags:
- analog-clock
- time-reading
- sim2real
---
# clockface
Reads an analog clock and returns the time. Finds the dial with a
COCO-pretrained detector, then reads the crop.
> **263703 reads 51.5% of 200 held-out real photographs to the minute**,
> against 49.0% for 263702. It is also wrong by more than half an hour
> more often -- 25.5% against 22.0% -- so this is a
> trade, not a clean win. Which one you want depends on whether a wrong answer
> costs you more than a missing one. Both are in this repository.
Version 263703, commit `1fce71d`. Reader backbone: `resnet50`.
## The two versions, same 200 photographs, same pipeline
| | to the minute | median | within 5 min | MAE | worse than 30 min |
|---|---|---|---|---|---|
| **263703** | **51.5%** | **1.00 min** | 69.0% | 40.4 min | 25.5% |
| 263702 | 49.0% | 1.25 min | **73.0%** | **34.0 min** | **22.0%** |
| guessing | ~0% | 180 min | 3% | 180 min | 97% |
The trade has a cause, and it is visible while training. Measured on real
photographs held out from training, epoch by epoch: readings correct to the
minute climb from 23.2% to 48.7% across all 22 epochs and never stop climbing,
while mean error bottoms out at epoch 8 and then gets worse. Training longer
makes the model sharper on the clocks it can read and less steady on the ones it
cannot. There is no epoch that is best at both, so the choice is a choice.
## What changed
**The reader gained a third head.** The other two each describe one hand, so a
dial whose hands are hard to tell apart corrupts both at once -- on 263702, 27.5%
of readings got the minute right and the hour wrong. The third head is asked for
the time itself over the whole 12-hour ring, coarsely, with no notion of a hand:
it cannot express a role confusion because it never assigns roles, and it only
has to break the twelve-way tie. In a controlled comparison -- same data, same
recipe, only the head differing -- it moved exactly what it was aimed at, hours
missed with the minute correct falling from 12.0% to 8.5%.
**The checkpoint is chosen on photographs, not renders.** Renders stop
predicting real accuracy about a third of the way through training; picking the
epoch renders liked cost 3 to 7 minutes of real error. It is chosen on
photographs held out from training and never in this test set, and by the
to-the-minute rate, because MAE and the to-the-minute rate disagree about which
epoch is best and only one of them is what this model is for.
## When to trust a reading
The reader ranks its own answers, and the ranking is worth more than the
headline. Declining the quarter it is least sure of takes it from
69.0% within five minutes to 96%.
```
200 readings. Keeping only the ones each signal is most sure of:
hand agreement
keep n MAE median <=5min
100% 200 40.40 1.00 69.0%
75% 150 26.06 1.00 80.0%
50% 100 15.16 0.75 86.0%
25% 50 10.31 0.50 90.0%
10% 20 7.20 0.75 95.0%
rank correlation with error +0.388 (useful)
head sharpness
keep n MAE median <=5min
100% 200 40.40 1.00 69.0%
75% 150 24.20 0.75 80.7%
50% 100 7.80 0.75 92.0%
25% 50 1.84 0.75 98.0%
10% 20 3.55 0.50 95.0%
rank correlation with error -0.504 (useful)
search margin
keep n MAE median <=5min
100% 200 40.40 1.00 69.0%
75% 150 21.12 0.75 82.0%
50% 100 10.33 0.75 91.0%
25% 50 5.75 0.50 96.0%
10% 20 0.57 0.50 100.0%
rank correlation with error -0.529 (useful)
A caution on reading the low-coverage rows: at 25% of 200 photographs a row
rests on 50 readings, so small differences between signals there are noise.
```
The signal this project set out to use for that was disagreement between the
hands. On weaker readers it ranked almost nothing; it becomes useful here, which
took a good enough model rather than a better signal.
## Stage 1
`fasterrcnn_mobilenet_v3_large_fpn` with COCO weights, on CPU, not fine-tuned.
It finds a clock in 93-96% of real photographs having never seen a render. A
dial locator trained on renders scored no better than not cropping at all and
was removed. On the 9 test photographs where stage 1 finds nothing, 1 is still
read within five minutes.
## Honest limits
The goal this was pointed at was reading the time to the minute 95% of the time.
It does not get there: 51.5% across all photographs, and about 90% even
when allowed to decline 85% of them.
The labels on these 200 photographs are third party. 106 were checked by hand
against the image; the rest were not. There is no first-party test set, and a
number measured on somebody else's labels is worth less than one measured on
your own.
## Licence
Apache 2.0. The synthetic dataset is CC BY 4.0.
|