Card: the test split's results, first on both TurnBench tasks
Browse files
README.md
CHANGED
|
@@ -17,7 +17,8 @@ tags:
|
|
| 17 |
|
| 18 |
Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for
|
| 19 |
[TurnBench](https://turnbench.sesame.com). It decides when a speaker's turn has ended, and whether a sound from the
|
| 20 |
-
other speaker is an interruption or an "mhm".
|
|
|
|
| 21 |
|
| 22 |
**Write-up:** <https://spero.patientdesk.ai/blog/kuroko>
|
| 23 |
|
|
@@ -87,11 +88,22 @@ The write-up gives the rules in full.
|
|
| 87 |
|
| 88 |
## Results
|
| 89 |
|
| 90 |
-
TurnBench
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
|
| 96 |
| | Recall | False-alarm rate | Median delay | Held-out recall | Held-out false-alarm rate |
|
| 97 |
|---|---|---|---|---|---|
|
|
@@ -136,14 +148,16 @@ read `wait_s`, `partner_s`, `deadline_s`, `resumed_s`, `held_s`, `confirm_s`, `p
|
|
| 136 |
|
| 137 |
## Limits
|
| 138 |
|
| 139 |
-
-
|
| 140 |
-
|
|
|
|
|
|
|
| 141 |
- TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice
|
| 142 |
agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer.
|
| 143 |
- A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim.
|
| 144 |
The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions
|
| 145 |
-
and backchannels. Kept at least 1 s apart, the events score 0.975 at 0.074 on ends of
|
| 146 |
-
interruptions.
|
| 147 |
- Studio audio with one clean channel per speaker, English only. Not measured on phone audio, with echo or as a live
|
| 148 |
stream.
|
| 149 |
|
|
|
|
| 17 |
|
| 18 |
Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for
|
| 19 |
[TurnBench](https://turnbench.sesame.com). It decides when a speaker's turn has ended, and whether a sound from the
|
| 20 |
+
other speaker is an interruption or an "mhm". On November 17, 2026 it had the highest recall of the 27 entries on
|
| 21 |
+
both tasks of TurnBench's test leaderboard.
|
| 22 |
|
| 23 |
**Write-up:** <https://spero.patientdesk.ai/blog/kuroko>
|
| 24 |
|
|
|
|
| 88 |
|
| 89 |
## Results
|
| 90 |
|
| 91 |
+
On TurnBench's test split (116 conversations, 6,293 ends of turn, 804 interruptions), scored by the benchmark's
|
| 92 |
+
maintainers, Kuroko has the highest recall of the 27 entries on both tasks, as the
|
| 93 |
+
[leaderboard](https://turnbench.sesame.com/models/kuroko) showed it on November 17, 2026:
|
| 94 |
+
|
| 95 |
+
| | Recall | False-alarm rate | Median delay | Rank by recall |
|
| 96 |
+
|---|---|---|---|---|
|
| 97 |
+
| End of turn | 0.983 | 0.065 | 619 ms | 1 of 27 |
|
| 98 |
+
| Interruption | 0.989 | 0.074 | 557 ms | 1 of 27 |
|
| 99 |
+
|
| 100 |
+
The lead is half a point on each task, about 30 ends of turn and about 4 interruptions. On ends of turn the next
|
| 101 |
+
entry, OpenYap Turn, has 0.978 recall and a median delay of 298 ms, half of Kuroko's.
|
| 102 |
+
|
| 103 |
+
The settings were chosen on the development split (38 conversations, 1,904 ends of turn, 347 interruptions) within a
|
| 104 |
+
false-alarm budget of 8%, so the scores there are optimistic. "Held out" is the same choice made on half of the
|
| 105 |
+
conversations and scored on the other half, over 500 random halves. That estimate was made before submitting. It
|
| 106 |
+
was within 0.001 of the test recall and off by 0.013 to 0.017 on the false-alarm rate.
|
| 107 |
|
| 108 |
| | Recall | False-alarm rate | Median delay | Held-out recall | Held-out false-alarm rate |
|
| 109 |
|---|---|---|---|---|---|
|
|
|
|
| 148 |
|
| 149 |
## Limits
|
| 150 |
|
| 151 |
+
- The lead on the test split is half a point of recall, from one submission. The leaderboard ranks by recall alone,
|
| 152 |
+
and on ends of turn the next entry is twice as fast.
|
| 153 |
+
- The development numbers come from 38 conversations, with settings chosen on them. Zero missed interruptions is an
|
| 154 |
+
in-sample number; on the test split recall is 0.989.
|
| 155 |
- TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice
|
| 156 |
agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer.
|
| 157 |
- A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim.
|
| 158 |
The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions
|
| 159 |
+
and backchannels. Kept at least 1 s apart, the events on the development split score 0.975 at 0.074 on ends of
|
| 160 |
+
turn and 1.000 at 0.070 on interruptions.
|
| 161 |
- Studio audio with one clean channel per speaker, English only. Not measured on phone audio, with echo or as a live
|
| 162 |
stream.
|
| 163 |
|