CDT5058 commited on
Commit
5b3166c
·
verified ·
1 Parent(s): 3ce67af

Updated README.md to have inference latency evals

Browse files
Files changed (1) hide show
  1. README.md +30 -0
README.md CHANGED
@@ -129,6 +129,36 @@ documents; it misses them in casual first-person writing.
129
  phi-redact v7 is essentially tied with Desert Ant's English composite at half
130
  the parameters and half the on-device footprint.
131
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
132
  ---
133
 
134
  ## Training data
 
129
  phi-redact v7 is essentially tied with Desert Ant's English composite at half
130
  the parameters and half the on-device footprint.
131
 
132
+ ### Inference latency
133
+
134
+ Measured 2026-10-03 against the live demo (corytrimm.com/phi-redact): the
135
+ same 15-template typing-length corpus as `evals/latency-bench/bench.py`,
136
+ 15 templates × 10 cycles = 150 redactions, timed from the page's own
137
+ per-redaction readout — TypeScript rules scan + ONNX detect + merge +
138
+ placeholders, end to end, onnxruntime-web WASM (Chromium on Linux).
139
+
140
+ | Input | p50 | p95 | p99 |
141
+ |---|---|---|---|
142
+ | Chat-length (150 samples) | **6.0 ms** | 8.0 ms | 9.0 ms |
143
+ | Long text (3,959 chars, 139 entities) | 572 ms | — | — |
144
+
145
+ Model load: 0.5 s cached, 4.4 s cold (11.3 MB int8 ONNX + runtime download).
146
+
147
+ Caveats: the model was cached for this run; browser version and hardware
148
+ details were not obtainable from the measurement setup (Linux VM, not a
149
+ phone); readout resolution is whole milliseconds. Not a head-to-head with
150
+ any vendor's published number — different hardware, corpus, and setup.
151
+ Methodology travels with the number; see `evals/latency-bench/`.
152
+
153
+ ### Synthetic redaction eval (164 cases)
154
+
155
+ `evals/synthetic-164/` carries a 164-case synthetic redaction test set, the
156
+ scoring harness, and v7's results: **0.812 full-value recall** (term recall
157
+ 0.615, precision 0.698, clean-case FP rate 0.120). Perfect 1.000 on
158
+ BANK_ACCOUNT, CREDIT_CARD, DRIVERS_LICENSE, EMAIL, PHONE, ROUTING_NUMBER;
159
+ weak on NAME (0.347) and ADDRESS (0.333). See the directory README for
160
+ methodology and the taxonomy note.
161
+
162
  ---
163
 
164
  ## Training data