kuroko / README.md
cloud0day3's picture
Describe the streaming step and the detector file
ee7308d verified
|
Raw History Blame Contribute Delete
13.3 kB
---
license: apache-2.0
language:
- en
pipeline_tag: voice-activity-detection
library_name: pytorch
tags:
- turn-taking
- end-of-turn
- interruption
- backchannel
- turnbench
- audio
---
# Kuroko
Kuroko is a turn-taking system for two-channel English conversation, built by PatientDesk AI for
[TurnBench](https://turnbench.sesame.com). It decides when a speaker's turn has ended, and whether a sound from the
other speaker is an interruption or an "mhm". On November 17, 2026 it had the highest recall of the 27 entries on
both tasks of TurnBench's test leaderboard.
**Write-up:** <https://spero.patientdesk.ai/blog/kuroko>
This repository holds Kuroko's weights, the one file needed to run them, its settings and its predictions for
TurnBench's development split, and the model as one streaming step for ONNX Runtime. The training code and the
training data are not public, and neither are the predictions for the test split, at the benchmark's request.
## Run the model
The model is a 14.9M-parameter causal transformer. It hears both speakers and answers 25 times a second for each
of them, from the past only.
```
pip install torch numpy scipy safetensors soundfile huggingface_hub
```
```python
import sys
import soundfile as sf
from huggingface_hub import snapshot_download
path = snapshot_download("cloud0day3/kuroko")
sys.path.insert(0, path)
import kuroko
model = kuroko.load(path)
audio, rate = sf.read("conversation.wav", dtype="int16") # (samples, 2): one speaker on each channel
answers = kuroko.answers(model, kuroko.features(audio, rate)) # (2, 11, n): speaker, answer, 25 a second
claims = answers[:, kuroko.ANSWERS.index("claim")]
```
`answers[s, a, k]` is a probability for speaker `s` that has heard the audio up to `(k + 1) / 25` seconds. The 11
answers are named in `kuroko.ANSWERS`:
| Answer | Meaning |
|---|---|
| `voice` | The speaker is making a sound |
| `claim` | That sound claims the floor. This is the answer the events are read from |
| `hold` | The speaker holds the floor, pauses included |
| 8 more | Whether the speaker, and whether the partner, will talk for half or more of 0 to 0.2 s, 0.2 to 0.6 s, 0.6 to 1.2 s and 1.2 to 2 s ahead |
The model was trained on 16-bit recordings at 48 kHz with one speaker per channel, brought to 16 kHz as
`kuroko.features` does it. Read that way, on the GPU the predictions in this repository were made on, the released
files give the answers behind them on all 38 dev conversations. On a CPU an answer differs by less than 0.001,
which can move a threshold crossing by one answer, 40 ms. Audio of more bits read as floats moves an answer by up
to 0.02.
Only `claim` has been measured against annotations. Offline, a recorded 12.6-minute conversation is answered in 11 s
on four CPU threads, in 40-second stretches. To hear a conversation while it happens, use the file under
*One step at a time*.
## From answers to events
The model is one of three parts, and this repository has the code of the model only.
1. **Detector.** [Silero VAD](https://github.com/snakers4/silero-vad) 6.2.3 runs on each channel in 32 ms steps and
marks stretches of sound. It is certain that a stretch has ended 0.26 s after its last sound.
2. **Rules.** The timing rules p99lab published for [turn-1-mini](https://github.com/P99Lab/turn-1-mini), without that
system's learned parts, with the numbers chosen again on the dev split (`policy.json`). Every stop opens a
candidate end of turn, which is committed once the partner has taken over or 2.45 s have passed, unless the
speaker is back.
3. **Model.** `claim` commits an interruption when it reaches 0.9, read from 0.32 s after a sound begins, before
the sound has lasted the 1.3 s the length rule needs. It also tells the end-of-turn rules that the partner has
taken over, when it reaches 0.5 for the partner's sound.
The write-up gives the rules in full.
## One step at a time
`onnx/kuroko-stream.onnx` is the same model cut to one step: 40 ms of both channels at 16 kHz in, the 11 answers
about the first channel out. It needs [ONNX Runtime](https://onnxruntime.ai) and nothing else, and it runs in a
browser. The mel frames are made inside it.
What the model has heard so far goes in beside the audio and comes back: the last frames, and each layer's keys and
values for the answers before. The caller keeps them, the newest answer written over the oldest:
```python
import numpy as np
import onnxruntime as ort
CONTEXT = 500 # answers remembered: 20 s
step = ort.InferenceSession("onnx/kuroko-stream.onnx")
turns = 1.0 / 10_000 ** (np.arange(32) / 32)
last = np.zeros((2, 320), np.float32) # the 20 ms before this step
heard = np.zeros((2, 4, 80), np.float32) # the frames before it
mask = np.full(CONTEXT, -1e30, np.float32) # 0 where a place in the memory holds an answer
keys = [np.zeros((6, 64, CONTEXT), np.float32) for _ in range(8)]
values = [np.zeros((6, CONTEXT, 64), np.float32) for _ in range(8)]
floor = np.full((2, 2, 80), 1e-10, np.float32)
def answer(k: int, audio: np.ndarray) -> np.ndarray:
"""The 11 answers about channel 0 after step k. `audio` is (2, 640): the next 40 ms of both channels."""
global last, heard
feeds = {"audio": np.concatenate([last, audio], axis=1), "floor": floor, "heard": heard, "mask": mask}
feeds |= {"cos": np.cos(k * turns).astype(np.float32), "sin": np.sin(k * turns).astype(np.float32)}
feeds |= {f"key{layer}": keys[layer] for layer in range(8)}
feeds |= {f"value{layer}": values[layer] for layer in range(8)}
answers, heard, made_keys, made_values = step.run(None, feeds)
place = k % CONTEXT
for layer in range(8):
keys[layer][:, :, place] = made_keys[layer]
values[layer][:, place] = made_values[layer]
mask[place] = 0.0
last = audio[:, -320:]
return answers
```
To answer about the other speaker, keep a second memory and give it the channels the other way round. `floor` is
added to each mel band's energy before the logarithm, and `1e-10` is what training used. The model was trained on
rooms. A channel that is digitally silent between its sounds, such as a synthetic voice, can be given a quiet
room by raising its floor; we have not scored that.
While nothing has been forgotten, a step's answers are the whole model's to within 0.002. After that the stream is
a different listener from the one that was entered, so it was scored by itself. On the development split, with the
same detector, rules and settings (recall / false-alarm rate / median delay):
| Heard as | End of turn | Interruption |
|---|---|---|
| 40-second stretches, as entered | 0.987 / 0.079 / 630 ms | 1.000 / 0.073 / 545 ms |
| One step at a time, 40 s remembered | 0.987 / 0.081 / 630 ms | 1.000 / 0.074 / 543 ms |
| 20 s remembered | 0.987 / 0.079 / 630 ms | 0.997 / 0.074 / 537 ms |
| 10 s remembered | 0.987 / 0.079 / 630 ms | 1.000 / 0.075 / 538 ms |
| 20 s remembered, weights as in this file | 0.987 / 0.079 / 630 ms | 0.997 / 0.074 / 537 ms |
The rows differ by at most one interruption of 347, two false alarms of 1,063 pauses and eight of 3,733 listener
sounds and noises. The file stores the weights in half precision, which makes it 31 MB and moved an answer by up
to 0.006 in the 15,000 we compared. In Chrome's WebAssembly, on one thread of an Apple M5 laptop, a step takes
about 3 ms.
`onnx/silero_vad.onnx` is the detector, Silero VAD 6.2.3, unchanged and under its own MIT license, so that a page
can fetch both files from one place.
## Results
On TurnBench's test split (116 conversations, 6,293 ends of turn, 804 interruptions), scored by the benchmark's
maintainers, Kuroko has the highest recall of the 27 entries on both tasks, as the
[leaderboard](https://turnbench.sesame.com/models/kuroko) showed it on November 17, 2026:
| | Recall | False-alarm rate | Median delay | Rank by recall |
|---|---|---|---|---|
| End of turn | 0.983 | 0.065 | 619 ms | 1 of 27 |
| Interruption | 0.989 | 0.074 | 557 ms | 1 of 27 |
The lead is half a point on each task, about 30 ends of turn and about 4 interruptions. On ends of turn the next
entry, OpenYap Turn, has 0.978 recall and a median delay of 298 ms, half of Kuroko's.
The settings were chosen on the development split (38 conversations, 1,904 ends of turn, 347 interruptions) within a
false-alarm budget of 8%, so the scores there are optimistic. "Held out" is the same choice made on half of the
conversations and scored on the other half, over 500 random halves. That estimate was made before submitting. It
was within 0.001 of the test recall and off by 0.013 to 0.017 on the false-alarm rate.
| | Recall | False-alarm rate | Median delay | Held-out recall | Held-out false-alarm rate |
|---|---|---|---|---|---|
| End of turn | 0.987 (1,879 of 1,904) | 0.079 (84 of 1,063) | 630 ms | 0.983 | 0.082 |
| Interruption | 1.000 (347 of 347) | 0.073 (272 of 3,733) | 545 ms | 0.988 | 0.061 |
To check these numbers, drop `predictions-dev.json` on the benchmark's [dev scorer](https://turnbench.sesame.com/dev).
It scores the file in the browser.
What each part adds, on the same split (recall / false-alarm rate / median delay):
| System | End of turn | Interruption |
|---|---|---|
| Rules, turn-1-mini's published settings, on our detector | 0.968 / 0.051 / 879 ms | 0.899 / 0.026 / 1,579 ms |
| Rules, settings chosen on dev | 0.983 / 0.080 / 692 ms | 0.977 / 0.076 / 1,078 ms |
| **Kuroko**: rules and the model | 0.987 / 0.079 / 630 ms | 1.000 / 0.073 / 545 ms |
## Training data
1,025 hours of two-speaker conversation: 786 hours of [Open Yap 1K](https://theagenticdatacompany.com/open-yap-1k)
and about 240 hours of PatientDesk's own recordings, made by contributors who agreed to this use. The labels come
from a timing rule applied in hindsight. None of the data is in this repository.
No audio and no label from TurnBench's dev or test split was trained on, and otoSpeech was not used. The dev split
chose the settings, checked the label rule and compared the systems above.
## Files
| File | Contents |
|---|---|
| `model.safetensors` | The weights, kept at step 12,000 |
| `config.json` | The model's shape |
| `kuroko.py` | The model, its features and how a conversation is answered. Nothing of the training |
| `policy.json` | The settings of the rules, in seconds and answer levels: `eot` for ends of turn, `int` for interruptions |
| `predictions-dev.json` | Kuroko's events for the 38 dev conversations, in TurnBench's submission format |
| `training-log.jsonl` | Training loss and the loss of each answer on held-out conversations, every 600 steps |
| `onnx/kuroko-stream.onnx` | The model as one 40 ms step for ONNX Runtime, weights in half precision |
| `onnx/silero_vad.onnx` | Silero VAD 6.2.3, the detector, with its license in `onnx/SILERO_VAD_LICENSE` |
Each block of `policy.json` is a full settings object, and each rule reads only part of it. The end-of-turn rules
read `wait_s`, `partner_s`, `deadline_s`, `resumed_s`, `held_s`, `confirm_s`, `partner_claim` and `claim_from_s` from
`eot`. The interruption rule reads `sustained_s`, `claim_fire`, `claim_from_s` and `claim_rise` from `int`.
`known_s` and `begun_s` are the detector's delays.
## Limits
- The lead on the test split is half a point of recall, from one submission. The leaderboard ranks by recall alone,
and on ends of turn the next entry is twice as fast.
- The development numbers come from 38 conversations, with settings chosen on them. Zero missed interruptions is an
in-sample number; on the test split recall is 0.989.
- TurnBench records two people, so the end-of-turn rules can wait for the second person to start talking. A voice
agent is the second speaker, and in an agent these rules would be left with the 2.45 s timer.
- A turn's end can be committed a second time, and an interruption is committed for every sound taken as a claim.
The scorer counts at most one false alarm per pause, and looks for interruptions only near annotated interruptions
and backchannels. Kept at least 1 s apart, the events on the development split score 0.975 at 0.074 on ends of
turn and 1.000 at 0.070 on interruptions.
- Studio audio with one clean channel per speaker, English only. The streaming step has been scored on the
development split's recordings only. Nothing has been measured on a live microphone, on phone audio or with echo.
The write-up has the rest: the label rule and its check against the annotators, the ablations, the causality test
and the credits.
## License
`model.safetensors`, `config.json`, `kuroko.py` and `onnx/kuroko-stream.onnx` are released under Apache-2.0.
`onnx/silero_vad.onnx` is Silero VAD, released by the Silero Team under the MIT license.
## Credits
TurnBench is by Sesame AI Labs, with the evaluation corpus from Mundo AI and the otoSpeech training set from Oto
([Jiang et al., 2026](https://arxiv.org/abs/2608.25218)). The timing rules are p99lab's. The detector is Silero VAD.
The model's future-activity targets follow voice activity projection
([Ekstedt and Skantze, 2022](https://arxiv.org/abs/2205.09812)).
Training data: The Agentic Data Company. Open Yap 1K: channel-separated English natural two-speaker conversations.
2026. Version 1.0. <https://theagenticdatacompany.com/open-yap-1k>