--- library_name: axllm license: apache-2.0 base_model: - Qwen/Qwen3-0.6B pipeline_tag: text-generation tags: - qwen3 - text-generation - axera - ax637 - axllm - w8a16 language: - en --- # Qwen3-0.6B on AXERA NPU Ready-to-run deployment package for `Qwen/Qwen3-0.6B` on AX637. - This release packages the AX637 text-only `axllm` runtime files. - The package includes the bundled `axllm` binary, compiled text `.axmodel` files, tokenizer data, and runtime configs. - The validated runtime layout uses `prefill_len=128`, `kv_cache_len=1023`, and `prefill_max_token_num=768`. - This package supports text-only chat through `axllm run` and the OpenAI-compatible `axllm serve` API. ## Supported Platform - [x] AX637 ## Validated Devices This package has been validated on the following device class: - AX637 development board ## Performance All measurements below were taken on AX637 with the packaged binary launched from the package root. `TTFT` stands for time to first token. The table below reports steady-state measurements and excludes the first request after startup. For this Qwen3 package, the benchmark prompts use the model's public `/no_think` instruction so that the measured outputs reflect direct answers or direct generation rather than a variable-length reasoning preamble. `Output tokens` below are model-token counts observed in the board-side runtime log. | Scenario | Input tokens | Output tokens | Prefill chunks | TTFT | Decode | |---|---:|---:|---:|---:|---:| | Short-answer text request (`/no_think` + city-name answer) | `28` | `5` | `1 x 128` | `234.47 ms` | `6.50 token/s avg` | | Long-output text generation (`/no_think` + number sequence request) | `30` | `256` | `1 x 128` | `233.89 ms` | `8.13 token/s avg` | For sustained decode throughput, use the long-output row as the more representative reference because the short-answer row terminates after only a few generated tokens. The packaged runtime uses the following context layout: - `prefill_len=128` - `kv_cache_len=1023` - `prefill_max_token_num=768` ## Startup Runtime Footprint | Item | Value | |---|---:| | `Package flash total` | `1.30 GiB` | | `Runtime CMM increment during board-side startup` | `774 MB` | The CMM value above was measured from the validated AX637 board-side startup log, where `remain_cmm` decreased from `2005 MB` after the first text layer init to `1231 MB` after post-model init. ## Package Layout ```text . ├── README.md ├── bin/ │ ├── axllm │ └── axllm.version.json ├── config.json ├── post_config.json ├── qwen3_tokenizer.txt ├── model.embed_tokens.weight.bfloat16.bin ├── qwen3_p128_l0_together.axmodel ├── ... ├── qwen3_p128_l27_together.axmodel └── qwen3_post.axmodel ``` This package uses a flat runtime layout. The bundled `axllm` binary lives under `bin/`, and the compiled text runtime files live at the repository root. ## Direct Inference with `axllm` ### Download the Model Package Download the release package from Hugging Face: ```shell mkdir -p AXERA-TECH/Qwen3-0.6B-AX637 cd AXERA-TECH/Qwen3-0.6B-AX637 hf download AXERA-TECH/Qwen3-0.6B-AX637 --local-dir . ``` ### Install `axllm` Option 1: use the bundled binary in this repository. ```bash chmod +x ./bin/axllm ``` If your shell does not already expose the AXERA runtime libraries and the bundled binary reports missing shared libraries such as `libax_engine.so`, add `/opt/lib` to `LD_LIBRARY_PATH` before launch. Option 2: install `axllm` from the public repository if you prefer a system-wide binary: ```shell git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git cd ax-llm ./install.sh ``` ### Run on the Board From the package root on the board: ```bash chmod +x ./bin/axllm ./bin/axllm serve . --port 8000 ``` Expected model id: ```text AXERA-TECH/Qwen3-0.6B-AX637 ``` Health check: ```bash curl http://127.0.0.1:8000/health curl http://127.0.0.1:8000/v1/models ``` Example output: ```json { "concurrency": 0, "max_concurrency": 1, "status": "healthy" } ``` ```json { "data": [ { "id": "AXERA-TECH/Qwen3-0.6B-AX637", "object": "model" } ], "object": "list" } ``` If you prefer the interactive CLI: ```bash chmod +x ./bin/axllm ./bin/axllm run . ``` ## Text Request The following request was validated on AX637 and returned the final answer successfully: ```bash curl http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "AXERA-TECH/Qwen3-0.6B-AX637", "messages": [ {"role": "user", "content": "/no_think\nWhat is the capital of the United States? Answer with the city name only."} ], "max_tokens": 32, "temperature": 0 }' ``` Example output: ```json { "choices": [ { "message": { "content": "Washington", "role": "assistant" } } ] } ``` Practical note: - By default, some prompts may emit a reasoning-style preamble before the final answer. - If you want direct-answer behavior for short factual prompts, prefix the last user message with `/no_think`, as shown in the validated example above. - If you do not use `/no_think`, keep enough `max_tokens` budget for both the reasoning span and the final answer. ## Conversion References If you need the original model files or want to rebuild the deployment artifacts, start with: - Original Hugging Face model: [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) - AXERA runtime repository: [AXERA-TECH/ax-llm](https://github.com/AXERA-TECH/ax-llm) ## Discussion - GitHub Issues: [AXERA-TECH/ax-llm/issues](https://github.com/AXERA-TECH/ax-llm/issues) - QQ group: `139953715`
