Kuroko
Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for TurnBench. It decides when a speaker's turn has ended, and whether a sound from the other speaker is an interruption or an "mhm". On November 17, 2026 it had the highest recall of the 27 entries on both tasks of TurnBench's test leaderboard.
Write-up: https://spero.patientdesk.ai/blog/kuroko
This repository holds Kuroko's weights, the one file needed to run them, its settings and its predictions for TurnBench's development split, and the model as one streaming step for ONNX Runtime. The training code and the training data are not public, and neither are the predictions for the test split, at the benchmark's request.
Run the model
The model is a 14.9M-parameter causal transformer. It hears both speakers and answers 25 times a second for each of them, from the past only.
pip install torch numpy scipy safetensors soundfile huggingface_hub
import sys
import soundfile as sf
from huggingface_hub import snapshot_download
path = snapshot_download("cloud0day3/kuroko")
sys.path.insert(0, path)
import kuroko
model = kuroko.load(path)
audio, rate = sf.read("conversation.wav", dtype="int16") # (samples, 2): one speaker on each channel
answers = kuroko.answers(model, kuroko.features(audio, rate)) # (2, 11, n): speaker, answer, 25 a second
claims = answers[:, kuroko.ANSWERS.index("claim")]
answers[s, a, k] is a probability for speaker s that has heard the audio up to (k + 1) / 25 seconds. The 11
answers are named in kuroko.ANSWERS:
| Answer | Meaning |
|---|---|
voice |
The speaker is making a sound |
claim |
That sound claims the floor. This is the answer the events are read from |
hold |
The speaker holds the floor, pauses included |
| 8 more | Whether the speaker, and whether the partner, will talk for half or more of 0 to 0.2 s, 0.2 to 0.6 s, 0.6 to 1.2 s and 1.2 to 2 s ahead |
The model was trained on 16-bit recordings at 48 kHz with one speaker per channel, brought to 16 kHz as
kuroko.features does it. Read that way, on the GPU the predictions in this repository were made on, the released
files give the answers behind them on all 38 dev conversations. On a CPU an answer differs by less than 0.001,
which can move a threshold crossing by one answer, 40 ms. Audio of more bits read as floats moves an answer by up
to 0.02.
Only claim has been measured against annotations. Offline, a recorded 12.6-minute conversation is answered in 11 s
on four CPU threads, in 40-second stretches. To hear a conversation while it happens, use the file under
One step at a time.
From answers to events
The model is one of three parts, and this repository has the code of the model only.
- Detector. Silero VAD 6.2.3 runs on each channel in 32 ms steps and marks stretches of sound. It is certain that a stretch has ended 0.26 s after its last sound.
- Rules. The timing rules p99lab published for turn-1-mini, without that
system's learned parts, with the numbers chosen again on the dev split (
policy.json). Every stop opens a candidate end of turn, which is committed once the partner has taken over or 2.45 s have passed, unless the speaker is back. - Model.
claimcommits an interruption when it reaches 0.9, read from 0.32 s after a sound begins, before the sound has lasted the 1.3 s the length rule needs. It also tells the end-of-turn rules that the partner has taken over, when it reaches 0.5 for the partner's sound.
The write-up gives the rules in full.
One step at a time
onnx/kuroko-stream.onnx is the same model cut to one step: 40 ms of both channels at 16 kHz in, the 11 answers
about the first channel out. It needs ONNX Runtime and nothing else, and it runs in a
browser. The mel frames are made inside it.
What the model has heard so far goes in beside the audio and comes back: the last frames, and each layer's keys and values for the answers before. The caller keeps them, the newest answer written over the oldest:
import numpy as np
import onnxruntime as ort
CONTEXT = 500 # answers remembered: 20 s
step = ort.InferenceSession("onnx/kuroko-stream.onnx")
turns = 1.0 / 10_000 ** (np.arange(32) / 32)
last = np.zeros((2, 320), np.float32) # the 20 ms before this step
heard = np.zeros((2, 4, 80), np.float32) # the frames before it
mask = np.full(CONTEXT, -1e30, np.float32) # 0 where a place in the memory holds an answer
keys = [np.zeros((6, 64, CONTEXT), np.float32) for _ in range(8)]
values = [np.zeros((6, CONTEXT, 64), np.float32) for _ in range(8)]
floor = np.full((2, 2, 80), 1e-10, np.float32)
def answer(k: int, audio: np.ndarray) -> np.ndarray:
"""The 11 answers about channel 0 after step k. `audio` is (2, 640): the next 40 ms of both channels."""
global last, heard
feeds = {"audio": np.concatenate([last, audio], axis=1), "floor": floor, "heard": heard, "mask": mask}
feeds |= {"cos": np.cos(k * turns).astype(np.float32), "sin": np.sin(k * turns).astype(np.float32)}
feeds |= {f"key{layer}": keys[layer] for layer in range(8)}
feeds |= {f"value{layer}": values[layer] for layer in range(8)}
answers, heard, made_keys, made_values = step.run(None, feeds)
place = k % CONTEXT
for layer in range(8):
keys[layer][:, :, place] = made_keys[layer]
values[layer][:, place] = made_values[layer]
mask[place] = 0.0
last = audio[:, -320:]
return answers
To answer about the other speaker, keep a second memory and give it the channels the other way round. floor is
added to each mel band's energy before the logarithm, and 1e-10 is what training used. The model was trained on
rooms. A channel that is digitally silent between its sounds, such as a synthetic voice, can be given a quiet
room by raising its floor; we have not scored that.
While nothing has been forgotten, a step's answers are the whole model's to within 0.002. After that the stream is a different listener from the one that was entered, so it was scored by itself. On the development split, with the same detector, rules and settings (recall / false-alarm rate / median delay):
| Heard as | End of turn | Interruption |
|---|---|---|
| 40-second stretches, as entered | 0.987 / 0.079 / 630 ms | 1.000 / 0.073 / 545 ms |
| One step at a time, 40 s remembered | 0.987 / 0.081 / 630 ms | 1.000 / 0.074 / 543 ms |
| 20 s remembered | 0.987 / 0.079 / 630 ms | 0.997 / 0.074 / 537 ms |
| 10 s remembered | 0.987 / 0.079 / 630 ms | 1.000 / 0.075 / 538 ms |
| 20 s remembered, weights as in this file | 0.987 / 0.079 / 630 ms | 0.997 / 0.074 / 537 ms |
The rows differ by at most one interruption of 347, two false alarms of 1,063 pauses and eight of 3,733 listener sounds and noises. The file stores the weights in half precision, which makes it 31 MB and moved an answer by up to 0.006 in the 15,000 we compared. In Chrome's WebAssembly, on one thread of an Apple M5 laptop, a step takes about 3 ms.
onnx/silero_vad.onnx is the detector, Silero VAD 6.2.3, unchanged and under its own MIT license, so that a page
can fetch both files from one place.
Results
On TurnBench's test split (116 conversations, 6,293 ends of turn, 804 interruptions), scored by the benchmark's maintainers, Kuroko has the highest recall of the 27 entries on both tasks, as the leaderboard showed it on November 17, 2026:
| Recall | False-alarm rate | Median delay | Rank by recall | |
|---|---|---|---|---|
| End of turn | 0.983 | 0.065 | 619 ms | 1 of 27 |
| Interruption | 0.989 | 0.074 | 557 ms | 1 of 27 |
The lead is half a point on each task, about 30 ends of turn and about 4 interruptions. On ends of turn the next entry, OpenYap Turn, has 0.978 recall and a median delay of 298 ms, half of Kuroko's.
The settings were chosen on the development split (38 conversations, 1,904 ends of turn, 347 interruptions) within a false-alarm budget of 8%, so the scores there are optimistic. "Held out" is the same choice made on half of the conversations and scored on the other half, over 500 random halves. That estimate was made before submitting. It was within 0.001 of the test recall and off by 0.013 to 0.017 on the false-alarm rate.
| Recall | False-alarm rate | Median delay | Held-out recall | Held-out false-alarm rate | |
|---|---|---|---|---|---|
| End of turn | 0.987 (1,879 of 1,904) | 0.079 (84 of 1,063) | 630 ms | 0.983 | 0.082 |
| Interruption | 1.000 (347 of 347) | 0.073 (272 of 3,733) | 545 ms | 0.988 | 0.061 |
To check these numbers, drop predictions-dev.json on the benchmark's dev scorer.
It scores the file in the browser.
What each part adds, on the same split (recall / false-alarm rate / median delay):
| System | End of turn | Interruption |
|---|---|---|
| Rules, turn-1-mini's published settings, on our detector | 0.968 / 0.051 / 879 ms | 0.899 / 0.026 / 1,579 ms |
| Rules, settings chosen on dev | 0.983 / 0.080 / 692 ms | 0.977 / 0.076 / 1,078 ms |
| Kuroko: rules and the model | 0.987 / 0.079 / 630 ms | 1.000 / 0.073 / 545 ms |
Training data
1,025 hours of two-speaker conversation: 786 hours of Open Yap 1K and about 240 hours of PatientDesk's own recordings, made by contributors who agreed to this use. The labels come from a timing rule applied in hindsight. None of the data is in this repository.
No audio and no label from TurnBench's dev or test split was trained on, and otoSpeech was not used. The dev split chose the settings, checked the label rule and compared the systems above.
Files
| File | Contents |
|---|---|
model.safetensors |
The weights, kept at step 12,000 |
config.json |
The model's shape |
kuroko.py |
The model, its features and how a conversation is answered. Nothing of the training |
policy.json |
The settings of the rules, in seconds and answer levels: eot for ends of turn, int for interruptions |
predictions-dev.json |
Kuroko's events for the 38 dev conversations, in TurnBench's submission format |
training-log.jsonl |
Training loss and the loss of each answer on held-out conversations, every 600 steps |
onnx/kuroko-stream.onnx |
The model as one 40 ms step for ONNX Runtime, weights in half precision |
onnx/silero_vad.onnx |
Silero VAD 6.2.3, the detector, with its license in onnx/SILERO_VAD_LICENSE |
Each block of policy.json is a full settings object, and each rule reads only part of it. The end-of-turn rules
read wait_s, partner_s, deadline_s, resumed_s, held_s, confirm_s, partner_claim and claim_from_s from
eot. The interruption rule reads sustained_s, claim_fire, claim_from_s and claim_rise from int.
known_s and begun_s are the detector's delays.
Limits
- The lead on the test split is half a point of recall, from one submission. The leaderboard ranks by recall alone, and on ends of turn the next entry is twice as fast.
- The development numbers come from 38 conversations, with settings chosen on them. Zero missed interruptions is an in-sample number; on the test split recall is 0.989.
- TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer.
- A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim. The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions and backchannels. Kept at least 1 s apart, the events on the development split score 0.975 at 0.074 on ends of turn and 1.000 at 0.070 on interruptions.
- Studio audio with one clean channel per speaker, English only. The streaming step has been scored on the development split's recordings only. Nothing has been measured on a live microphone, on phone audio or with echo.
The write-up has the rest: the label rule and its check against the annotators, the ablations, the causality test and the credits.
License
model.safetensors, config.json, kuroko.py and onnx/kuroko-stream.onnx are released under Apache-2.0.
onnx/silero_vad.onnx is Silero VAD, released by the Silero Team under the MIT license.
Credits
TurnBench is by Sesame AI Labs, with the evaluation corpus from Mundo AI and the otoSpeech training set from Oto (Jiang et al., 2026). The timing rules are p99lab's. The detector is Silero VAD. The model's future-activity targets follow voice activity projection (Ekstedt and Skantze, 2022).
Training data: The Agentic Data Company. Open Yap 1K: channel-separated English natural two-speaker conversations. 2026. Version 1.0. https://theagenticdatacompany.com/open-yap-1k
- Downloads last month
- 86