Qwen3.5 9B PARO

--- library_name: transformers license: apache-2.0 pipeline_tag: image-text-to-text base_model: - Qwen/Qwen3.5-9B tags: - mlx --- # z-lab/Qwen3.5-9B-PARO **Pairwise Rotation Quantization for Efficient Reasoning LLM Inference** <p> <a href="https://arxiv.org/abs/2511.10645"><img src="https://img.shields.io/badge/arXiv-2511.10645-b31b1b.svg" alt="Paper"></a> <a href="https://paroquant.z-lab.ai"><img src="https://img.shields.io/badge/Blog-ParoQuant-blue" alt="Blog"></a> <a href="https://huggingface.co/collections/z-lab/paroquant"><img src="https://img.shields.io/badge/%F0%9F%A4%97-Models-yellow" alt="Models"></a> <a href="https://pypi.org/project/paroquant/"><img src="https://img.shields.io/pypi/v/paroquant" alt="PyPI"></a> </p> ParoQuant is the state-of-the-art INT4 quantization for LLMs. It closes the accuracy gap with FP16 while running at near-AWQ speed. Supports NVIDIA GPUs (vLLM, Transformers) and Apple Silicon (MLX). For more information, see https://github.com/z-lab/paroquant. z-lab/Qwen3.5-9B-PARO is a 4-bit [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) quantized with ParoQuant. Check out other ParoQuant models from the Hugging Face [collection](https://huggingface.co/collections/z-lab/paroquant). ## Quick Start ### Installation ```bash # NVIDIA GPU (CUDA 12.9) pip install "paroquant[vllm]" # NVIDIA GPU (CUDA 13.0) pip install "paroquant[vllm]" "vllm==0.19.1" \ --extra-index-url https://wheels.vllm.ai/0.19.1/cu130 \ --extra-index-url https://download.pytorch.org/whl/cu130 # Apple Silicon pip install "paroquant[mlx]" ``` ### Interactive Chat ```bash python -m paroquant.cli.chat --model z-lab/Qwen3.5-9B-PARO ``` ### OpenAI-Compatible API Server For vLLM, you can directly use `vllm serve` to serve ParoQuant models: ```bash vllm serve z-lab/Qwen3.5-9B-PARO --port 8000 ``` For other frameworks: ```bash python -m paroquant.cli.serve --model z-lab/Qwen3.5-9B-PARO --port 8000 ``` For MLX, add `--vlm` if you wish to load the VLM components and use the model's multimodal features. For vLLM, VLM components are loaded by default and can be skipped with the server argument `--language-model-only`. > [!NOTE] > The visual components in this checkpoint is stored in original precision, and only the language components are quantized to 4 bits; as a result, the model size is larger than a fully-quantized model. Avoid loading the V...

Source context: 35018 downloads · 52 likes · Pipeline image-text-to-text · Library transformers · Repo z-lab/Qwen3.5-9B-PARO

Source context: 17177 downloads · 47 likes · Pipeline image-text-to-text · Library transformers · Repo z-lab/Qwen3.5-9B-PARO