Canary 1b v2 Sherpa Onnx Int8

canary-1b-v2 -- sherpa-onnx int8 ONNX export

An sherpa-onnx offline-recognizer package for nvidia/canary-1b-v2, in the same encoder/decoder/tokens layout sherpa-onnx already ships for canary-180m-flash. Canary is an attention encoder-decoder model, not a transducer, so there is no joiner file.

Base model / attribution

• Base model: nvidia/canary-1b-v2 by NVIDIA, licensed under CC-BY-4.0.
• Architecture: FastConformer encoder (32 layers) + Transformer decoder (8 layers), 978M parameters, unified SentencePiece tokenizer (16,384 tokens), 25 European languages.
• This derivative is also distributed under CC-BY-4.0, as required by the base model's license.

Changes made

This repo is a community conversion, not an official NVIDIA or k2-fsa release. Starting from the .nemo checkpoint above, the following was done:

1. Exported to ONNX (encoder + decoder graphs) using an adaptation of k2-fsa/sherpa-onnx's official export_onnx_180m_flash.py script, retargeted at nvidia/canary-1b-v2 instead of nvidia/canary-180m-flash (export_canary_1b_v2.py, environment: torch 2.6.0 CPU, nemo_toolkit[asr] 2.7.3, onnx 1.17.0, onnxruntime 1.17.1).
2. Quantized to int8 (dynamic quantization, onnxruntime.quantization.quantize_dynamic).
3. Repackaged into the sherpa-onnx offline-Canary layout: encoder.int8.onnx, decoder.int8.onnx, tokens.txt (no joiner), plus a test_wavs/ folder -- same structure as the official sherpa-onnx-nemo-canary-180m-flash-en-es-de-fr-int8.tar.bz2 release.
4. ONNX metadata was set explicitly: num_decoder_layers = 10 and decoder_hidden_size = 1024. Note the semantics: the sherpa-onnx C++ runtime uses num_decoder_layers as the COUNT of decoder_mems_list_* inputs it feeds. NeMo's TransformerDecoder keeps num_layers + 2 memory states when a final_layer_norm exists (canary-1b-v2: 8 layers + final LN = 10). The runtime's default of 6 matches 180m-flash, not this model.

Key non-obvious fixes needed vs. the 180m script (why k2-fsa/sherpa-onnx#1767 stayed unresolved):

• Decoder must be exported with dynamo=True. The decoder wrapper reads the decode position via decoder_input_ids[0][-1].item(); the legacy TorchScript exporter bakes that .item() in as a constant 0, freezing the positional embedding -- the model then emits one correct token and degenerates into repeating its own prompt. The dynamo exporter keeps it a dynamic value.
• Distinct dynamic-axis names for decoder_input_ids vs. the mems (torch 2.6's exporter rejects the 180m script's shared "num_tokens" symbol as a constraint violation).
• tokens.txt built with tokenizer.id_to_piece(i), not ids_to_text([i]): canary-1b-v2's CanaryBPETokenizer strips the leading space on single-id detokenization, which would drop the ▁ word-boundary markers (output would have no spaces between words).
• NeMo 2.7.3's TransformerDecoder.forward(return_mems=True) returns a 2-tuple (mems, xatt_scores); older NeMo returned just the list.
• dynamo emits IR version 10; onnxruntime 1.17.1 (and the ORT bundled in many sherpa-onnx builds) rejects IR > 9, so the decoder is downgraded to IR 9 before and after quantization.

Verified with the sherpa-onnx (1.13.4) Python runtime on the bundled test_wavs/: German ASR ("Alles hat ein Ende, nur die Wurst hat zwei."), English ASR, and de<->en translation all decode correctly.

• Date of conversion: 2026-07-11.

Files

File	Description
sherpa-onnx-nemo-canary-1b-v2-int8.tar.bz2	Full package: encoder, decoder, tokens, test_wavs/
encoder.int8.onnx	int8-quantized encoder (also inside the tar)
decoder.int8.onnx	int8-quantized decoder (also inside the tar)
encoder.fp16.onnx	fp16 encoder (see "fp16 variant" below)
decoder.fp16.onnx	fp16 decoder (see "fp16 variant" below)
tokens.txt	Vocabulary (also inside the tar)
sha256sums.txt	Checksums, for verifying your download

fp16 variant

encoder.fp16.onnx / decoder.fp16.onnx are half-precision versions of the same export (use the same tokens.txt). For CPU inference the int8 files are the recommended choice (smaller, comparable speed); fp16 is mainly interesting for CUDA execution providers.

Producing an fp16 pair that current sherpa-onnx actually loads required more than a plain convert_float_to_float16 call, documented here for anyone reproducing this:

1. Converted with onnxruntime's maintained float16 converter (onnxruntime.transformers.float16, keep_io_types=True so the I/O stays float32), after file-based shape inference (infer_shapes_path) on the

   2 GB encoder so cast nodes get inserted at fp16/fp32 boundaries.

2. Stripped the stale value_info entries from the decoder afterwards (the converter leaves entries typed float32 that contradict the retyped LayerNormalization outputs; onnxruntime rejects the graph otherwise).
3. Ran an offline BASIC-level onnxruntime optimization pass (constant folding, no fusions) over both graphs. Without this, the ORT version bundled in current sherpa-onnx crashes in its SimplifiedLayerNormFusion pass on inserted casts attached to Constant outputs shared across the 32 encoder layers ("Attempting to get index by a name which does not exist: InsertedPrecisionFreeCast_...").

Verified with sherpa-onnx 1.13.4 (Python) on the bundled test_wavs/: German and English ASR both decode correctly from the fp16 pair.

Usage (sherpa-onnx)

Python API:

import sherpa_onnx

recognizer = sherpa_onnx.OfflineRecognizer.from_nemo_canary(
    encoder="./sherpa-onnx-nemo-canary-1b-v2-int8/encoder.int8.onnx",
    decoder="./sherpa-onnx-nemo-canary-1b-v2-int8/decoder.int8.onnx",
    tokens="./sherpa-onnx-nemo-canary-1b-v2-int8/tokens.txt",
)

Or via the sherpa-onnx-ws-* / sherpa-onnx-offline CLI binaries with:

--canary-encoder=./sherpa-onnx-nemo-canary-1b-v2-int8/encoder.int8.onnx
--canary-decoder=./sherpa-onnx-nemo-canary-1b-v2-int8/decoder.int8.onnx
--tokens=./sherpa-onnx-nemo-canary-1b-v2-int8/tokens.txt
--canary-src-lang=en
--canary-tgt-lang=en

Requires sherpa-onnx >= the release that added Canary C++ runtime support (PR #2352, merged 2025-07-07) -- any sherpa-onnx release since then supports this model.

Not an official release

This is a community-produced ONNX conversion, uploaded to the maintainer's own HF account. It is not published, reviewed, or endorsed by NVIDIA or k2-fsa. k2-fsa's official asr-models release tag does not currently include a canary-1b-v2 package (see k2-fsa/sherpa-onnx#1767).