Dualturn Endpointing

DualTurn Endpointing

A small, causal, audio-only turn-taking model for voice agents. It listens to both channels of a live conversation (user + agent) and emits, frame-by-frame at 12.5 Hz, the signals a voice agent needs to decide when to speak — with end-of-turn (endpointing) as the primary output.

Adapted from the DualTurn paper (arXiv:2603.08216) and fine-tuned for the endpointing head. It runs on frozen Mimi features, adds only ~1.5M parameters on top of the encoder, needs no ASR and no cloud, and is designed to run on-device / on CPU.

How it's meant to be used

The model sits inside a live two-channel call between a user (CH0) and the agent (CH1). It continuously listens to both channels at the same time and, every 80 ms, emits per-frame predictions: VAD and FVAD for both speakers, and the user's end-of-turn (EOT). A voice agent consumes these — chiefly the user's EOT — to decide when it's the agent's turn to speak (and VAD/FVAD to sense who's talking and who's about to talk). The agent channel is not just context: hearing the agent's own speech (and overlaps/interruptions) is what makes the user-EOT judgment reliable in a real conversation.

It can also run as a single-stream (user-only) detector: feed the user audio and silence the agent channel (pass mono, or a zeros agent channel). You still get the user's EOT + user VAD/FVAD; you simply lose the agent-context benefit. This is the setup used for benchmarking and for user-mic-only deployments.

Output	Shape	Channels	Meaning
eot_probs	[1, T, 1]	user	P(user has finished their turn) — the endpointing signal
vad_probs	[1, T, 2]	user, agent	P(speaking)
fvad_probs	[1, T, 2, 4]	user, agent	P(channel speaks within a future window) — 4 horizons: 0–240 / 240–640 / 640–1200 / 1200–2000 ms

Frames are 80 ms (12.5 Hz); T ≈ audio_seconds × 12.5. Streaming/causal — outputs at time t use only audio up to t.

Inference modes & latency

Four ready-to-use paths (all fp32, same weights). Offline processes a whole file at once; streaming runs per 80 ms tick for live calls. ONNX paths need only onnxruntime (no PyTorch).

#	mode	runtime	latency (CPU, 4 threads)	deps
1	Offline	PyTorch — model(wav, sr)	~7.5 ms/frame (batch, ~11× real-time)	torch + transformers
2	Streaming	PyTorch — model.streaming()	~63 ms / 80 ms tick (real-time, flat)	torch + transformers
3	Offline	ONNX — model.onnx	~7.3 ms/frame (~11× real-time)	onnxruntime
4	Streaming	ONNX — stream_tick.onnx	~52 ms / 80 ms tick (real-time, flat, tightest tail)	onnxruntime

Offline latency is amortized batch throughput (all frames returned after the file is processed); streaming latency is the real per-tick cost in a live call — both well under the 80 ms budget. Streaming is causal + bounded (transformer KV capped at 250 frames = 10 s), so per-tick latency stays flat over arbitrarily long calls. Streaming outputs are decision-equivalent to offline — they match the batch model to within ~0.02–0.03 (fp32 op-order drift in the incremental path), which doesn't move threshold+sustain endpoint decisions.

Inference interval (compute ↔ latency)

Two streaming ONNX graphs are shipped — pick by how often you need to react vs how much CPU you want to spend. Both are strictly causal (the transformer runs frame-by-frame inside the graph — no future audio) and decision-equivalent to offline — they track the batch model to within ~0.02–0.03 (fp32 op-order drift in the streaming path; the two graphs are bit-identical to each other, Δ≈0). The 240 ms graph runs one shared encoder pass over 3 frames instead of three separate passes, so it's ~1.8× cheaper per second of audio:

• stream_tick.onnx — 1 frame per call ([2, 1920] = 80 ms).
• stream_tick_240.onnx — 3 frames per call ([2, 5760] = 240 ms), returns all 3 × 80 ms frames.

Measured on a real call (CPU, 4 threads, carried KV cache). Grace = idle time left in the interval after inference (interval − latency); CPU busy = per-stream duty cycle, fraction of wall-clock the inference runs to keep up with one live stream (latency × calls/sec):

graph	infer every	latency / call	calls / audio-sec	grace / call	CPU busy / stream
stream_tick.onnx	80 ms	52 ms	12.5	28 ms	~65%
stream_tick_240.onnx	240 ms	85 ms	4.17	155 ms	~36%

• 80 ms — snappiest, but heaviest. Each call eats 52 ms → only 28 ms grace, pool busy ~65% per stream. Fastest reactions, least room for anything else.
• 240 ms — much cheaper, lots of slack. Each call takes 85 ms (vs 3 × 52 = 156 ms for three 80 ms ticks) but leaves 155 ms grace and sits at ~36% busy.
• The real win is at scale. CPU per second of audio drops ~1.8× (650 → 354 ms/s, ≈45% less), so one machine holds roughly ~1.8× as many concurrent calls — the per-stream % looks modest, but that factor multiplies across every simultaneous call in your fleet. (CPU busy % is a single-stream duty cycle measured with 4 threads — read it as relative cost between graphs, not absolute single-core load.)
• Accuracy is unaffected — the 240 ms graph is decision-equivalent to the 80 ms one (endpoint decisions match); the only cost is that you learn the turn ended at the next 240 ms boundary (up to one interval later).

1. Offline — PyTorch

import torch, torchaudio
from transformers import AutoModel

model = AutoModel.from_pretrained("anyreach-ai/dualturn-endpointing", trust_remote_code=True).eval()

wav, sr = torchaudio.load("conversation.wav")   # [2, T]  CH0 = user, CH1 = agent
with torch.no_grad():
    out = model(wav, sr=sr)

out.eot_probs    # [1, T, 1]      P(user end-of-turn)
out.vad_probs    # [1, T, 2]      P(speaking) — (user, agent)
out.fvad_probs   # [1, T, 2, 4]   P(speaks soon) — (user, agent) × (0-240/240-640/640-1200/1200-2000 ms)

Turn the end-of-turn stream into a decision with a thin policy — e.g. fire when eot_probs stays above a threshold for a short sustain window:

eot = out.eot_probs[0, :, 0]
thr, sustain = 0.5, 3                       # 3 frames ≈ 240 ms
fired = (eot > thr).unfold(0, sustain, 1).all(-1).nonzero()   # frame indices of detected turn-ends

Mono input [T] or [1, T] is also accepted (treated as user with a silent agent — a single-stream fallback). Any sample rate works (sr=); audio is internally resampled to 24 kHz (Mimi's input rate). The call above is the offline mode (whole file at once). For live calls, use the streaming mode below.

2. Streaming — PyTorch (real-time)

For live calls, don't re-run the whole file each tick — use the stateful streamer. It's causal (uses only past audio), keeps bounded state (transformer KV window + LSTM state), and runs ~63 ms per 80 ms tick on CPU (≥4 threads), flat regardless of call length.

from transformers import AutoModel
model = AutoModel.from_pretrained("anyreach-ai/dualturn-endpointing", trust_remote_code=True).eval()
streamer = model.streaming()

# feed audio as it arrives, in 80 ms chunks: [2, 1920] @ 24 kHz  (CH0=user, CH1=agent)
for chunk in chunks_at_24kHz:            # mono [1920] is also accepted (silent agent)
    out = streamer.push(chunk)           # None until the next 12.5 Hz frame is ready
    if out is not None:
        p_eot = out.eot_probs[0, -1, 0].item()      # latest P(user end-of-turn)
        # fire your end-of-turn policy on p_eot (threshold + short sustain)

streamer.reset()                         # between calls

push returns a DualTurnOutput (same eot_probs / vad_probs / fvad_probs) for the new frame(s). It matches the offline model to ~3e-4 mean H1. Feed 24 kHz (resample the continuous stream once, not per-chunk, to avoid chunk-edge artifacts).

3. Offline — ONNX (no PyTorch)

A self-contained model.onnx (Mimi encoder + readout in one graph) is included for CPU / on-device deployment with just onnxruntime. Input is a 2-channel waveform at 24 kHz [2, T] (CH0=user, CH1=agent); resample first with any tool.

import numpy as np, librosa, onnxruntime as ort
from huggingface_hub import hf_hub_download

onnx_path = hf_hub_download("anyreach-ai/dualturn-endpointing", "model.onnx")
sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])

wav, sr = librosa.load("conversation.wav", sr=24000, mono=False)   # [2, T] @ 24 kHz, CH0=user CH1=agent
eot, vad, fvad = sess.run(None, {"audio_24k": wav.astype("float32")})
# eot  [1, T', 1]      P(user end-of-turn)
# vad  [1, T', 2]      P(speaking) — (user, agent)
# fvad [1, T', 2, 4]   P(speaks soon) — (user, agent) × (0-240/240-640/640-1200/1200-2000 ms)

eot = eot[0, :, 0]
fired = np.where(np.convolve((eot > 0.5).astype(int), np.ones(3), "valid") == 3)[0]   # sustained ≥240 ms

Runs comfortably faster than real-time on CPU (≈1 s of compute per 10 s of audio, single-threaded). The graph is fully self-contained — no torch or transformers needed at inference.

4. Streaming — ONNX (real-time, no PyTorch)

Torch-free real-time streaming via stream_tick.onnx + the onnx_streaming.py helper (DualTurnONNXStreamer). Each 80 ms tick only computes the new frame — the graph carries all state (conv buffer, transformer KV window, downsample history, LSTM state) in/out, so latency is low (~52 ms/tick) and flat over long calls. Decision-equivalent to the PyTorch model.

from onnx_streaming import DualTurnONNXStreamer     # download onnx_streaming.py + stream_tick.onnx from the repo

streamer = DualTurnONNXStreamer()                    # auto-downloads stream_tick.onnx (via huggingface_hub)
for chunk in chunks_24k:                             # each: [2, 1920] float32 @ 24 kHz (CH0=user, CH1=agent; mono OK)
    out = streamer.push(chunk)
    p_eot = out["eot"]                               # P(user end-of-turn); also out["vad"] [2], out["fvad"] [2,4]
    # fire your end-of-turn policy on p_eot (threshold + short sustain)
streamer.reset()                                     # between calls

Needs only onnxruntime + numpy (+ huggingface_hub for the auto-download). Feed 24 kHz audio in exactly 1920-sample (80 ms) chunks.

To spend ~1.8× less CPU (at the cost of reacting every 240 ms instead of 80 ms), use the 240 ms graph stream_tick_240.onnx via DualTurnONNXStreamer240 — same helper file, [2, 5760] chunks, returns the 3 × 80 ms frames per call. Both graphs are strictly causal and decision-equivalent to offline; see Inference interval.

from onnx_streaming import DualTurnONNXStreamer240      # downloads stream_tick_240.onnx

streamer = DualTurnONNXStreamer240()
for chunk in chunks_240ms:                               # each: [2, 5760] float32 @ 24 kHz
    for fr in streamer.push(chunk):                      # list of 3 frames: fr["eot"], fr["vad"], fr["fvad"]
        ...                                              # apply your end-of-turn policy on fr["eot"]
streamer.reset()

Performance

Benchmarked on LiveKit's eot-bench (a neutral end-of-turn benchmark). Metric is a latency / false-cutoff Pareto: FC@N ms = % of mid-turn pauses falsely fired...