cloud0day3 commited on
Commit
1ecabfb
·
verified ·
1 Parent(s): 879acfa

Card: the test split's results, first on both TurnBench tasks

Browse files
Files changed (1) hide show
  1. README.md +24 -10
README.md CHANGED
@@ -17,7 +17,8 @@ tags:
17
 
18
  Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for
19
  [TurnBench](https://turnbench.sesame.com). It decides when a speaker's turn has ended, and whether a sound from the
20
- other speaker is an interruption or an "mhm".
 
21
 
22
  **Write-up:** <https://spero.patientdesk.ai/blog/kuroko>
23
 
@@ -87,11 +88,22 @@ The write-up gives the rules in full.
87
 
88
  ## Results
89
 
90
- TurnBench development split: 38 conversations, 1,904 ends of turn, 347 interruptions, scored by the benchmark's
91
- scorer. The settings were chosen on this split within a false-alarm budget of 8%. "Held out" is the same choice made
92
- on half of the conversations and scored on the other half, over 500 random halves. Kuroko was submitted to TurnBench
93
- on November 13, 2026. The test split is scored by the benchmark's maintainers, and its results are theirs to
94
- publish on the [leaderboard](https://turnbench.sesame.com).
 
 
 
 
 
 
 
 
 
 
 
95
 
96
  | | Recall | False-alarm rate | Median delay | Held-out recall | Held-out false-alarm rate |
97
  |---|---|---|---|---|---|
@@ -136,14 +148,16 @@ read `wait_s`, `partner_s`, `deadline_s`, `resumed_s`, `held_s`, `confirm_s`, `p
136
 
137
  ## Limits
138
 
139
- - These are development numbers on 38 conversations, with settings chosen on them. Zero missed interruptions is an
140
- in-sample number; held out, recall is 0.988.
 
 
141
  - TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice
142
  agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer.
143
  - A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim.
144
  The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions
145
- and backchannels. Kept at least 1 s apart, the events score 0.975 at 0.074 on ends of turn and 1.000 at 0.070 on
146
- interruptions.
147
  - Studio audio with one clean channel per speaker, English only. Not measured on phone audio, with echo or as a live
148
  stream.
149
 
 
17
 
18
  Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for
19
  [TurnBench](https://turnbench.sesame.com). It decides when a speaker's turn has ended, and whether a sound from the
20
+ other speaker is an interruption or an "mhm". On November 17, 2026 it had the highest recall of the 27 entries on
21
+ both tasks of TurnBench's test leaderboard.
22
 
23
  **Write-up:** <https://spero.patientdesk.ai/blog/kuroko>
24
 
 
88
 
89
  ## Results
90
 
91
+ On TurnBench's test split (116 conversations, 6,293 ends of turn, 804 interruptions), scored by the benchmark's
92
+ maintainers, Kuroko has the highest recall of the 27 entries on both tasks, as the
93
+ [leaderboard](https://turnbench.sesame.com/models/kuroko) showed it on November 17, 2026:
94
+
95
+ | | Recall | False-alarm rate | Median delay | Rank by recall |
96
+ |---|---|---|---|---|
97
+ | End of turn | 0.983 | 0.065 | 619 ms | 1 of 27 |
98
+ | Interruption | 0.989 | 0.074 | 557 ms | 1 of 27 |
99
+
100
+ The lead is half a point on each task, about 30 ends of turn and about 4 interruptions. On ends of turn the next
101
+ entry, OpenYap Turn, has 0.978 recall and a median delay of 298 ms, half of Kuroko's.
102
+
103
+ The settings were chosen on the development split (38 conversations, 1,904 ends of turn, 347 interruptions) within a
104
+ false-alarm budget of 8%, so the scores there are optimistic. "Held out" is the same choice made on half of the
105
+ conversations and scored on the other half, over 500 random halves. That estimate was made before submitting. It
106
+ was within 0.001 of the test recall and off by 0.013 to 0.017 on the false-alarm rate.
107
 
108
  | | Recall | False-alarm rate | Median delay | Held-out recall | Held-out false-alarm rate |
109
  |---|---|---|---|---|---|
 
148
 
149
  ## Limits
150
 
151
+ - The lead on the test split is half a point of recall, from one submission. The leaderboard ranks by recall alone,
152
+ and on ends of turn the next entry is twice as fast.
153
+ - The development numbers come from 38 conversations, with settings chosen on them. Zero missed interruptions is an
154
+ in-sample number; on the test split recall is 0.989.
155
  - TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice
156
  agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer.
157
  - A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim.
158
  The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions
159
+ and backchannels. Kept at least 1 s apart, the events on the development split score 0.975 at 0.074 on ends of
160
+ turn and 1.000 at 0.070 on interruptions.
161
  - Studio audio with one clean channel per speaker, English only. Not measured on phone audio, with echo or as a live
162
  stream.
163