# Qwen3-ASR-0.6B int4-AWQ — v0.8.0 Deploy Bundle TensorRT-Edge-LLM v0.8.0 ASR engine bundle for NVIDIA Jetson (Orin-class). Mirrors the layout of `qwen3-tts-base-v080-deploy`. ## Precision | Component | Precision | |----------------|------------------| | LLM backbone | **int4-AWQ (W4A16)** | | audio encoder | **fp16** | | KV cache | fp16 | ## Contents / engine sizes + md5 ``` jv-workers/ libNvInfer_edgellm_plugin.so 45,338,696 B md5 17385ff85ff6e22da258eca31863867b models/qwen3-asr-base/engines/ llm/ llm.engine 550,819,820 B md5 4f3496c3b1d2790eb5c3a4a2fe1ba465 embedding.safetensors 311,165,016 B md5 8db9ceda288e2470694a0bc33dbfd381 tokenizer.json 11,429,653 B md5 68e0da75e29b5190b2b98c2e8a49d2e2 tokenizer_config.json, processed_chat_template.json, config.json audio/ audio_encoder.engine 381,216,588 B md5 f7a7fa8cb2b31d20e38cf664528ef03e config.json ``` ## Validated metrics 10-clip primed set (`primed_results.json`), greedy decode: - Chinese: CER **0%** (6/6 exact modulo trailing punctuation) - English: WER **~11%** (one substitution: "drop it down" -> "Drape it down") - all clips `finish_reason: end-of-sequence` (no runaway / no empty output) ## ⚠️ rope_type must be `mrope` The LLM `engines/llm/config.json` here carries `rope_scaling.rope_type = "mrope"` (mrope_section `[24,20,20]`). This is a load-bearing fix over the exported default (`"linear"`). With `linear`, the int4 engine emits garbage (repetition loops / empty). Do NOT regress it to `linear`. (`config.json.bak`, if present, is the pre-fix broken `linear` variant — delete it; do not deploy it.) ## ⚠️ Production decode contract (REQUIRED for correct output) The bare engine WITHOUT priming produces garbage. You MUST drive it with: - **Sampling:** `temperature=0.0`, `top_k=1`, `top_p=1.0` (greedy) - `apply_chat_template=true`, `add_generation_prompt=false` - **Assistant-turn prime** — request supplies a *partial assistant turn*: ```json { "batch_size": 1, "temperature": 0.0, "top_k": 1, "top_p": 1.0, "max_generate_length": 256, "apply_chat_template": true, "add_generation_prompt": false, "requests": [{ "messages": [ { "role": "system", "content": "" }, { "role": "user", "content": [{ "type": "audio", "audio": "/path/to/mel.safetensors" }] }, { "role": "assistant", "content": "language English " } ]}] } ``` `"language "` where ` ` ∈ {Chinese, English, ...}. - **Post-process:** strip the leading `"language "` from `output_text` (production `stripLanguagePrefix`). ## Run command (llm_inference reference) Production ASR runs through `llm_inference` / the `qwen3_asr_worker`. Direct reference invocation using the v0.8.0 `llm_inference` binary: ```bash export EDGELLM_PLUGIN_PATH=/path/to/jv-workers/libNvInfer_edgellm_plugin.so export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH # binary lives in the build tree: edgellm-v080-build/build/examples/llm/llm_inference ./llm_inference \ --engineDir models/qwen3-asr-base/engines/llm \ --multimodalEngineDir models/qwen3-asr-base/engines/audio \ --inputFile input_asr.json \ --outputFile output_asr.json ``` NOTE: `--multimodalEngineDir` points at the PARENT directory that contains the audio engine + its `config.json` (here `engines/audio`), NOT at the `.engine` file. Mel input is produced by `tensorrt-edgellm-preprocess-audio`. ## Provenance Quantization command, engine-build flags, the `quantize.py:737` `extra_state_dict` (modelopt>=0.39) export call, and the rope fix are recorded in `/home/harvest/project/qwen3-asr-ab/SOURCE_PROVENANCE.md`.
