Whisperdrz

--- license: mit library_name: whisperdrz pipeline_tag: automatic-speech-recognition tags: - whisper - speaker-diarization - diarization - speech-recognition - asr - word-timestamps language: - en --- # WhisperDRZ WhisperDRZ is a speaker-aware automatic speech recognition model. It transcribes audio into text with **word-level timestamps**, **per-line speaker tags**, and **non-speech event tags** ([laugh], [breath], ...). It is a Whisper-style encoder-decoder model and handles long audio by chunking and stitching internally. - 🎙️ **Try it in the browser** — no install: **[fluxions.ai/transcribe](https://fluxions.ai/transcribe)** - 💻 **Code (inference)**: <https://github.com/fluxions-ai/whisperdrz> - 📖 **Full write-up** — approach, honest results, what didn't work: see the repo's `writeup.md` ## Usage ```bash pip install git+https://github.com/fluxions-ai/whisperdrz ``` Requires Python 3.12+ and FFmpeg. Runs on a CUDA GPU or on CPU. Weights download automatically from this repo on first use. ### Command line ```bash whisperdrz audio.wav # defaults: this model, --lang en whisperdrz audio.wav --output_format json > out.json whisperdrz audio.wav --lang auto # auto-detect language ``` ### Python ```python import whisperdrz from whisperdrz.audio import load_audio, SAMPLE_RATE transcriber = whisperdrz.load_model("whisperdrz-large-v3.safetensors", lang="en") audio, _ = load_audio("audio.wav", sample_rate=SAMPLE_RATE) result = transcriber.transcribe(audio.mean(0)) # mono, 16 kHz print(result.text) # speaker-tagged text with timestamps print(result.segments) # list of {speaker, start, end, text} ``` You can also point `load_model` at this repo id (`fluxions/whisperdrz`) directly. ## Output format Each line begins with a speaker tag. Timed words and tags are wrapped in a start/end timestamp pair; the first and last word of each line are always timed: ``` [0] <|0.00|>Hello<|0.45|> there <|0.80|>world.<|1.10|> [1] <|1.20|>Hi<|1.40|> <|1.45|>[laugh]<|1.60|> <|1.70|>there.<|1.95|> ``` - `[0]`, `[1]`, ... are speaker IDs; `[c]` marks crowd/ambient. - `<|t|>` are timestamps in seconds (two decimals), always in a pair wrapping a word or tag. - `[laugh]`, `[breath]`, and similar are non-speech event tags. ## Evaluation Measured on this checkpoint: | Benchmark | WER | DER (miss / FA / conf) | cpWER / tcpWER | |---|---|---|---| | ESB (English AS...