Qwen3.6 27B W4A16

--- license: apache-2.0 base_model: Qwen/Qwen3.6-27B base_model_relation: quantized tags: - quantized - compressed-tensors - vllm - qwen3 - qwen3.6 - dense - vision - image-text-to-text - multimodal - 4-bit - w4a16 - int4 - post-training-quantization - long-context - 262k-context - gated-deltanet - linear-attention - tool-use - reasoning - agentic - safetensors - sglang - text-generation-inference - 88plug - autoround - llmcompressor - thinking - qwen language: - en pipeline_tag: image-text-to-text library_name: transformers model_type: qwen3_5 --- ## Load path (important) These weights are **compressed-tensors** (`pack-quantized` / `int-quantized`). | Runtime | Supported | |---------|-----------| | **vLLM ≥ 0.21** | **Yes — preferred** (auto-detect CT; no `--quantization` flag) | | transformers + `compressed-tensors` | Yes for many text models; multimodal may need custom code | | Text Generation Inference (TGI) | **Not supported** for these CT packs | | Hugging Face Inference Widget | Often fails — use vLLM locally instead | ```bash # Preferred vllm serve 88plug/<ModelName> --trust-remote-code ``` Do **not** deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors. # Qwen3.6-27B-W4A16 **~14 GB on disk. Full 262k native context on a single RTX 4090 or 24 GB GPU.** W4A16 post-training quantization of [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — a **vision-language model** (images + text → text) with a hybrid Gated-DeltaNet + dense-MLP architecture. BF16 baseline is ~54 GB. AutoRound (iters=200) with G32 group size. All dense Linear layers (attention + MLP) quantized to W4A16-G32. Gated DeltaNet recurrent layers excluded entirely (BF16) for vLLM correctness. > **Vision calibration note:** Calibration corpus is text-only. The vision encoder receives W4A16 quantization with text-derived calibration signal only. Text quality targets are fully met; vision inference quality may be reduced relative to text. For vision-critical workloads, consider the W8A16 variant. --- ## At a Glance | Property | Value | |---|---| | Base model | `Qwen/Qwen3.6-27B` | | Release tier | Gold (AutoRound iters=200) | | Quant method | AutoRound W4A16-G32 iters=200 | | FLAC status | Not measured (T+7d milestone) | | Architecture | Hybrid Gated-DeltaNet + Den...