Aller au contenu
ModelPrésentéExterne
Browse files

Sortformer Diarization CoreML

CoreML port of NVIDIA Sortformer for end-to-end streaming speaker diarization on Apple Silicon.

Sourcehuggingface.co/Alkd/Sortformer-Diarization-CoreML/tree/b4929a8a32a9d56d9a36ff1f0f0c252f4e5d374a
TypeOtherVersionRevision b4929a8CClasse

Source du modèle

Type
Other
Taille du fichier
108 B–220 MB
Version
vRevision b4929a8
Mis à jour
22 sept. 2026
Téléchargements
5
Source
Hugging Face

Description de la source

Sortformer Diarization (CoreML)

CoreML port of NVIDIA Sortformer for end-to-end streaming speaker diarization on Apple Silicon.

Runs on the Neural Engine via CoreML. No separate embedding extraction or clustering — the model directly predicts per-frame speaker activity for up to 4 speakers.

Model Details

  • Architecture: Sortformer (Sort Loss + 17-layer FastConformer encoder + 18-layer Transformer)
  • Base model: nvidia/diar_streaming_sortformer_4spk-v2.1 (117M params)
  • Task: Speaker diarization (up to 4 speakers)
  • Input: 128-dim log-mel features, streamed in chunks
  • Output: Per-frame speaker activity probabilities (sigmoid)
  • Format: CoreML .mlmodelc (compiled pipeline, 2 sub-models)
  • Size: ~230 MB

Streaming Configuration

ParameterValue
Sample rate16 kHz
Mel bins128
n_fft400
Hop length160
Chunk length6s
Left context1 chunk
Right context7 chunks
Subsampling factor8
Speaker cache length188 frames
FIFO length40 frames
Max speakers4

Input/Output Shapes

Inputs:

TensorShapeDescription
chunk[1, 112, 128]Mel features for current chunk
chunk_lengths[1]Valid frames in chunk
spkcache[1, 188, 512]Speaker cache state
spkcache_lengths[1]Valid entries in speaker cache
fifo[1, 40, 512]FIFO buffer state
fifo_lengths[1]Valid entries in FIFO

Outputs:

TensorShapeDescription
speaker_preds_out[1, 242, 4]Speaker activity probabilities
chunk_pre_encoder_embs_out[1, 14, 512]Chunk embeddings for state update
chunk_pre_encoder_lengths_out[1]Valid embedding frames

Usage

Used by speech-swift for speaker diarization:

audio diarize meeting.wav --engine sortformer

let diarizer = try await SortformerDiarizer.fromPretrained()
let result = diarizer.diarize(audio: samples, sampleRate: 16000)
for segment in result.segments {
    print("Speaker \(segment.speakerId): \(segment.startTime)s - \(segment.endTime)s")
}

Pipeline Architecture

The model is a CoreML pipeline with two sub-models:

  1. PreEncoder (model0) — Runs pre_encode on the mel chunk, concatenates with speaker cache and FIFO state
  2. Head (model1) — Full FastConformer encoder + Transformer + sigmoid speaker heads

State management (FIFO rotation, speaker cache compression) is handled in Swift outside the model.

License

CC-BY-4.0

Sources

1 source
Hugging FacePrincipalVersion b4929a8a32a9d56d9a36ff1f0f0c252f4e5d374a
huggingface.co/Alkd/Sortformer-Diarization-CoreML/tree/b4929a8a32a9d56d9a36ff1f0f0c252f4e5d374a

Vérifié 6 sept.

Download files

Artefacts du modèle

8 artefacts

Sortformer.mlmodelc/analytics/coremldata.bin

bin · 202 B · SHA-256 e1a18b8f5119…6539 · Hugging Face

Download file

Sortformer.mlmodelc/coremldata.bin

bin · 1,07 KB · SHA-256 29f0563d086d…9992 · Hugging Face

Download file

Sortformer.mlmodelc/model0/analytics/coremldata.bin

bin · 108 B · SHA-256 5a8281049b2a…e58a · Hugging Face

Download file

Sortformer.mlmodelc/model0/coremldata.bin

bin · 640 B · SHA-256 1221dd5d8951…86e4 · Hugging Face

Download file

Sortformer.mlmodelc/model0/weights/0-weight.bin

bin · 8,53 MB · SHA-256 88a98803e351…2fa4 · Hugging Face

Download file

Sortformer.mlmodelc/model1/analytics/coremldata.bin

bin · 108 B · SHA-256 5a8281049b2a…e58a · Hugging Face

Download file

Sortformer.mlmodelc/model1/coremldata.bin

bin · 605 B · SHA-256 efacb38511f5…aba3 · Hugging Face

Download file

Sortformer.mlmodelc/model1/weights/1-weight.bin

bin · 220 MB · SHA-256 4c85926af776…f1db · Hugging Face

Download file

Extraits de sources

2 extraits
Readme6 sept.huggingface.co/Alkd/Sortformer-Diarization-CoreML

--- license: cc-by-4.0 tags: - speaker-diarization - coreml - apple-silicon - neural-engine - sortformer datasets: - voxconverse language: - en pipeline_tag: audio-classification --- # Sortformer Diarization (CoreML) CoreML port of [NVIDIA Sortformer](https://arxiv.org/abs/2409.06656) for end-to-end streaming speaker diarization on Apple Silicon. Runs on the Neural Engine via CoreML. No separate embedding extraction or clustering — the model directly predicts per-frame speaker activity for up to 4 speakers. ## Model Details - **Architecture**: Sortformer (Sort Loss + 17-layer FastConformer encoder + 18-layer Transformer) - **Base model**: `nvidia/diar_streaming_sortformer_4spk-v2.1` (117M params) - **Task**: Speaker diarization (up to 4 speakers) - **Input**: 128-dim log-mel features, streamed in chunks - **Output**: Per-frame speaker activity probabilities (sigmoid) - **Format**: CoreML `.mlmodelc` (compiled pipeline, 2 sub-models) - **Size**: ~230 MB ## Streaming Configuration | Parameter | Value | |-----------|-------| | Sample rate | 16 kHz | | Mel bins | 128 | | n_fft | 400 | | Hop length | 160 | | Chunk length | 6s | | Left context | 1 chunk | | Right context | 7 chunks | | Subsampling factor | 8 | | Speaker cache length | 188 frames | | FIFO length | 40 frames | | Max speakers | 4 | ## Input/Output Shapes **Inputs:** | Tensor | Shape | Description | |--------|-------|-------------| | `chunk` | `[1, 112, 128]` | Mel features for current chunk | | `chunk_lengths` | `[1]` | Valid frames in chunk | | `spkcache` | `[1, 188, 512]` | Speaker cache state | | `spkcache_lengths` | `[1]` | Valid entries in speaker cache | | `fifo` | `[1, 40, 512]` | FIFO buffer state | | `fifo_lengths` | `[1]` | Valid entries in FIFO | **Outputs:** | Tensor | Shape | Description | |--------|-------|-------------| | `speaker_preds_out` | `[1, 242, 4]` | Speaker activity probabilities | | `chunk_pre_encoder_embs_out` | `[1, 14, 512]` | Chunk embeddings for state update | | `chunk_pre_encoder_lengths_out` | `[1]` | Valid embedding frames | ## Usage Used by [speech-swift](https://github.com/soniqo/speech-swift) for speaker diarization: ```bash audio diarize meeting.wav --engine sortformer ``` ```swift let diarizer = try await SortformerDiarizer.fromPretrained() let result = diarizer.diarize(audio: samples, sampleRate: 16000) for segment in result.segments {...

Note6 sept.huggingface.co/Alkd/Sortformer-Diarization-CoreML

Source context: 5 downloads · 2 likes · Pipeline audio-classification · Repo Alkd/Sortformer-Diarization-CoreML