TurnBench
A multi-domain benchmark for evaluating conversational turn-taking. We hand-annotate end-of-turn and interruption events in dual-channel human conversations, and measure how accurately and how quickly models detect them.
Leaderboard
Models evaluated against TurnBench's held-out test set, ranked by recall. The ideal model is capable of high recall while maintaining a low false-positive rate and low latency.
- Recall:
- share of true events detected (higher is better).
- FPR:
- false-positive rate on test (lower is better); above 0.15 does not qualify.
- Latency:
- median delay after the event, p50 in ms (lower is better).
Dataset
TurnBench evaluates models on a 30-hour corpus of studio-recorded, dual-channel dyadic speech. The corpus is provided by Mundo AI and features 154 dialogues, 106 actors, and is balanced across 6 conversation types1. It is also split evenly between clean and noisy recording conditions. Every dialogue is labeled by three independent annotators (Fleiss's κ = 0.76) where the ground truth label is derived from 2/3 consensus.
Additionally, we include a training dataset, hand-labeled under the same protocol as the evaluation set and release it alongside the benchmark. All three splits are available on Hugging Face.
- train
- 104 hotoSpeech
collected by Oto
audio + annotations - dev
- 7.3 hturn-benchmark-dev
collected by Mundo AI
audio + annotations - test
- 22.9 hturn-benchmark-test
collected by Mundo AI
audio only
Evaluation
Per conversation, TurnBench requires models to output a discrete list of times2 for the following events:
- End-of-turn
- When a speaker has finished speaking and the floor is open for the other speaker to take.FP: declaring end-of-turn during a mid-turn pause.
- Interruption
- When a speaker takes the floor while the other is still talking.FP: declaring an interruption during a backchannel or because of noise, channel bleed, or echo.