Kuroko

Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for TurnBench. It decides when a speaker's turn has ended, and whether a sound from the other speaker is an interruption or an "mhm". On November 17, 2026 it had the highest recall of the 27 entries on both tasks of TurnBench's test leaderboard.

Write-up: https://spero.patientdesk.ai/blog/kuroko

This repository holds Kuroko's weights, the one file needed to run them, its settings and its predictions for TurnBench's development split, and the model as one streaming step for ONNX Runtime. The training code and the training data are not public, and neither are the predictions for the test split, at the benchmark's request.

Run the model

The model is a 14.9M-parameter causal transformer. It hears both speakers and answers 25 times a second for each of them, from the past only.

pip install torch numpy scipy safetensors soundfile huggingface_hub
import sys

import soundfile as sf
from huggingface_hub import snapshot_download

path = snapshot_download("cloud0day3/kuroko")
sys.path.insert(0, path)
import kuroko

model = kuroko.load(path)
audio, rate = sf.read("conversation.wav", dtype="int16")  # (samples, 2): one speaker on each channel
answers = kuroko.answers(model, kuroko.features(audio, rate))  # (2, 11, n): speaker, answer, 25 a second
claims = answers[:, kuroko.ANSWERS.index("claim")]

answers[s, a, k] is a probability for speaker s that has heard the audio up to (k + 1) / 25 seconds. The 11 answers are named in kuroko.ANSWERS:

Answer Meaning
voice The speaker is making a sound
claim That sound claims the floor. This is the answer the events are read from
hold The speaker holds the floor, pauses included
8 more Whether the speaker, and whether the partner, will talk for half or more of 0 to 0.2 s, 0.2 to 0.6 s, 0.6 to 1.2 s and 1.2 to 2 s ahead

The model was trained on 16-bit recordings at 48 kHz with one speaker per channel, brought to 16 kHz as kuroko.features does it. Read that way, on the GPU the predictions in this repository were made on, the released files give the answers behind them on all 38 dev conversations. On a CPU an answer differs by less than 0.001, which can move a threshold crossing by one answer, 40 ms. Audio of more bits read as floats moves an answer by up to 0.02.

Only claim has been measured against annotations. Offline, a recorded 12.6-minute conversation is answered in 11 s on four CPU threads, in 40-second stretches. To hear a conversation while it happens, use the file under One step at a time.

From answers to events

The model is one of three parts, and this repository has the code of the model only.

  1. Detector. Silero VAD 6.2.3 runs on each channel in 32 ms steps and marks stretches of sound. It is certain that a stretch has ended 0.26 s after its last sound.
  2. Rules. The timing rules p99lab published for turn-1-mini, without that system's learned parts, with the numbers chosen again on the dev split (policy.json). Every stop opens a candidate end of turn, which is committed once the partner has taken over or 2.45 s have passed, unless the speaker is back.
  3. Model. claim commits an interruption when it reaches 0.9, read from 0.32 s after a sound begins, before the sound has lasted the 1.3 s the length rule needs. It also tells the end-of-turn rules that the partner has taken over, when it reaches 0.5 for the partner's sound.

The write-up gives the rules in full.

One step at a time

onnx/kuroko-stream.onnx is the same model cut to one step: 40 ms of both channels at 16 kHz in, the 11 answers about the first channel out. It needs ONNX Runtime and nothing else, and it runs in a browser. The mel frames are made inside it.

What the model has heard so far goes in beside the audio and comes back: the last frames, and each layer's keys and values for the answers before. The caller keeps them, the newest answer written over the oldest:

import numpy as np
import onnxruntime as ort

CONTEXT = 500  # answers remembered: 20 s
step = ort.InferenceSession("onnx/kuroko-stream.onnx")
turns = 1.0 / 10_000 ** (np.arange(32) / 32)
last = np.zeros((2, 320), np.float32)  # the 20 ms before this step
heard = np.zeros((2, 4, 80), np.float32)  # the frames before it
mask = np.full(CONTEXT, -1e30, np.float32)  # 0 where a place in the memory holds an answer
keys = [np.zeros((6, 64, CONTEXT), np.float32) for _ in range(8)]
values = [np.zeros((6, CONTEXT, 64), np.float32) for _ in range(8)]
floor = np.full((2, 2, 80), 1e-10, np.float32)


def answer(k: int, audio: np.ndarray) -> np.ndarray:
    """The 11 answers about channel 0 after step k. `audio` is (2, 640): the next 40 ms of both channels."""
    global last, heard
    feeds = {"audio": np.concatenate([last, audio], axis=1), "floor": floor, "heard": heard, "mask": mask}
    feeds |= {"cos": np.cos(k * turns).astype(np.float32), "sin": np.sin(k * turns).astype(np.float32)}
    feeds |= {f"key{layer}": keys[layer] for layer in range(8)}
    feeds |= {f"value{layer}": values[layer] for layer in range(8)}
    answers, heard, made_keys, made_values = step.run(None, feeds)
    place = k % CONTEXT
    for layer in range(8):
        keys[layer][:, :, place] = made_keys[layer]
        values[layer][:, place] = made_values[layer]
    mask[place] = 0.0
    last = audio[:, -320:]
    return answers

To answer about the other speaker, keep a second memory and give it the channels the other way round. floor is added to each mel band's energy before the logarithm, and 1e-10 is what training used. The model was trained on rooms. A channel that is digitally silent between its sounds, such as a synthetic voice, can be given a quiet room by raising its floor; we have not scored that.

While nothing has been forgotten, a step's answers are the whole model's to within 0.002. After that the stream is a different listener from the one that was entered, so it was scored by itself. On the development split, with the same detector, rules and settings (recall / false-alarm rate / median delay):

Heard as End of turn Interruption
40-second stretches, as entered 0.987 / 0.079 / 630 ms 1.000 / 0.073 / 545 ms
One step at a time, 40 s remembered 0.987 / 0.081 / 630 ms 1.000 / 0.074 / 543 ms
20 s remembered 0.987 / 0.079 / 630 ms 0.997 / 0.074 / 537 ms
10 s remembered 0.987 / 0.079 / 630 ms 1.000 / 0.075 / 538 ms
20 s remembered, weights as in this file 0.987 / 0.079 / 630 ms 0.997 / 0.074 / 537 ms

The rows differ by at most one interruption of 347, two false alarms of 1,063 pauses and eight of 3,733 listener sounds and noises. The file stores the weights in half precision, which makes it 31 MB and moved an answer by up to 0.006 in the 15,000 we compared. In Chrome's WebAssembly, on one thread of an Apple M5 laptop, a step takes about 3 ms.

onnx/silero_vad.onnx is the detector, Silero VAD 6.2.3, unchanged and under its own MIT license, so that a page can fetch both files from one place.

Results

On TurnBench's test split (116 conversations, 6,293 ends of turn, 804 interruptions), scored by the benchmark's maintainers, Kuroko has the highest recall of the 27 entries on both tasks, as the leaderboard showed it on November 17, 2026:

Recall False-alarm rate Median delay Rank by recall
End of turn 0.983 0.065 619 ms 1 of 27
Interruption 0.989 0.074 557 ms 1 of 27

The lead is half a point on each task, about 30 ends of turn and about 4 interruptions. On ends of turn the next entry, OpenYap Turn, has 0.978 recall and a median delay of 298 ms, half of Kuroko's.

The settings were chosen on the development split (38 conversations, 1,904 ends of turn, 347 interruptions) within a false-alarm budget of 8%, so the scores there are optimistic. "Held out" is the same choice made on half of the conversations and scored on the other half, over 500 random halves. That estimate was made before submitting. It was within 0.001 of the test recall and off by 0.013 to 0.017 on the false-alarm rate.

Recall False-alarm rate Median delay Held-out recall Held-out false-alarm rate
End of turn 0.987 (1,879 of 1,904) 0.079 (84 of 1,063) 630 ms 0.983 0.082
Interruption 1.000 (347 of 347) 0.073 (272 of 3,733) 545 ms 0.988 0.061

To check these numbers, drop predictions-dev.json on the benchmark's dev scorer. It scores the file in the browser.

What each part adds, on the same split (recall / false-alarm rate / median delay):

System End of turn Interruption
Rules, turn-1-mini's published settings, on our detector 0.968 / 0.051 / 879 ms 0.899 / 0.026 / 1,579 ms
Rules, settings chosen on dev 0.983 / 0.080 / 692 ms 0.977 / 0.076 / 1,078 ms
Kuroko: rules and the model 0.987 / 0.079 / 630 ms 1.000 / 0.073 / 545 ms

Training data

1,025 hours of two-speaker conversation: 786 hours of Open Yap 1K and about 240 hours of PatientDesk's own recordings, made by contributors who agreed to this use. The labels come from a timing rule applied in hindsight. None of the data is in this repository.

No audio and no label from TurnBench's dev or test split was trained on, and otoSpeech was not used. The dev split chose the settings, checked the label rule and compared the systems above.

Files

File Contents
model.safetensors The weights, kept at step 12,000
config.json The model's shape
kuroko.py The model, its features and how a conversation is answered. Nothing of the training
policy.json The settings of the rules, in seconds and answer levels: eot for ends of turn, int for interruptions
predictions-dev.json Kuroko's events for the 38 dev conversations, in TurnBench's submission format
training-log.jsonl Training loss and the loss of each answer on held-out conversations, every 600 steps
onnx/kuroko-stream.onnx The model as one 40 ms step for ONNX Runtime, weights in half precision
onnx/silero_vad.onnx Silero VAD 6.2.3, the detector, with its license in onnx/SILERO_VAD_LICENSE

Each block of policy.json is a full settings object, and each rule reads only part of it. The end-of-turn rules read wait_s, partner_s, deadline_s, resumed_s, held_s, confirm_s, partner_claim and claim_from_s from eot. The interruption rule reads sustained_s, claim_fire, claim_from_s and claim_rise from int. known_s and begun_s are the detector's delays.

Limits

  • The lead on the test split is half a point of recall, from one submission. The leaderboard ranks by recall alone, and on ends of turn the next entry is twice as fast.
  • The development numbers come from 38 conversations, with settings chosen on them. Zero missed interruptions is an in-sample number; on the test split recall is 0.989.
  • TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer.
  • A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim. The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions and backchannels. Kept at least 1 s apart, the events on the development split score 0.975 at 0.074 on ends of turn and 1.000 at 0.070 on interruptions.
  • Studio audio with one clean channel per speaker, English only. The streaming step has been scored on the development split's recordings only. Nothing has been measured on a live microphone, on phone audio or with echo.

The write-up has the rest: the label rule and its check against the annotators, the ablations, the causality test and the credits.

License

model.safetensors, config.json, kuroko.py and onnx/kuroko-stream.onnx are released under Apache-2.0. onnx/silero_vad.onnx is Silero VAD, released by the Silero Team under the MIT license.

Credits

TurnBench is by Sesame AI Labs, with the evaluation corpus from Mundo AI and the otoSpeech training set from Oto (Jiang et al., 2026). The timing rules are p99lab's. The detector is Silero VAD. The model's future-activity targets follow voice activity projection (Ekstedt and Skantze, 2022).

Training data: The Agentic Data Company. Open Yap 1K: channel-separated English natural two-speaker conversations. 2026. Version 1.0. https://theagenticdatacompany.com/open-yap-1k

Downloads last month
86
Safetensors
Model size
14.9M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Papers for cloud0day3/kuroko