Dualturn Qwen2.5 Mimi 0.5B

--- license: apache-2.0 tags: - turn-taking - voice-activity-detection - conversation - audio - speech - qwen2 - mimi - onnx datasets: - anyreach-ai/dualturn-switchboard-turn-taking - anyreach-ai/dualturn-otospeech-turn-taking language: - en pipeline_tag: audio-classification --- # DualTurn-Qwen2.5-Mimi-0.5B Real-time dual-channel turn-taking model. Given stereo audio (user mic + agent speaker), predicts per-frame: | Output | Shape | Description | |--------|-------|-------------| | `vad_probs` | [B, T, 2] | P(speaking now) — `[:,0]`=user `[:,1]`=agent | | `fvad_probs` | [B, T, 8] | P(future speech) at 240/480/960/2000 ms — user `0:4`, agent `4:8` | | `eot_probs` | [B, T, 2] | P(end of turn) per channel | | `bot_probs` | [B, T, 2] | P(beginning of turn) per channel | | `hold_probs` | [B, T, 2] | P(within-turn hold/pause) per channel | | `bc_probs` | [B, T, 2] | P(backchannel) per channel | Frame rate: **12.5 Hz** (80 ms per frame). Audio is resampled internally to 24 kHz. <img src="https://arxiv.org/html/2603.08216v1/arc-vertical.drawio.png" width="70%" /> --- ## FP32 Inference ```bash pip install transformers torch torchaudio safetensors ``` ```python import torch, torchaudio from transformers import AutoModel model = AutoModel.from_pretrained( "anyreach-ai/dualturn-qwen2.5-mimi-0.5B", trust_remote_code=True, ) model.eval() wav, sr = torchaudio.load("conversation.wav") # [2, T] CH0=user CH1=agent with torch.no_grad(): out = model(wav, sr=sr) print(out.vad_probs.shape) # [1, T, 2] print(out.fvad_probs.shape) # [1, T, 8] print(out.eot_probs.shape) # [1, T, 2] ``` --- ## ONNX Inference (CPU, KV Cache) The ONNX checkpoint runs the Qwen backbone with a rolling KV cache. Pass empty `past_key_values` for full-context batch inference, or maintain the cache across steps for low-latency streaming (~41 ms/step at 5 s context). ```bash pip install onnxruntime transformers torch torchaudio safetensors huggingface_hub ``` ```python import numpy as np, torch, torch.nn as nn, torchaudio, os from transformers import MimiModel from safetensors.torch import load_file from huggingface_hub import hf_hub_download, snapshot_download import onnxruntime as ort N_LAYERS, N_HEADS, HEAD_DIM, D = 24, 2, 64, 896 # Mimi encoder mimi = MimiModel.from_pretrained("kyutai/mimi").eval() @torch.no_grad() def encode(wav_1d): x = wav_1d.unsqueeze(0).unsqueez...

Source context: 871 downloads · 2 likes · Pipeline audio-classification · Repo anyreach-ai/dualturn-qwen2.5-mimi-0.5B