Qwen3.6 27B W8A16

--- license: apache-2.0 base_model: Qwen/Qwen3.6-27B base_model_relation: quantized language: - en tags: - qwen3 - qwen - dense - vision - image-text-to-text - multimodal - quantized - int8 - INT8 - w8a16 - compressed-tensors - vllm - autoround - llmcompressor - long-context - 262k-context - gated-deltanet - linear-attention - 8-bit - ptq - safetensors - sglang - 88plug - post-training-quantization - thinking - reasoning pipeline_tag: image-text-to-text library_name: transformers model_type: qwen3_5 --- ## Load path (important) These weights are **compressed-tensors** (`pack-quantized` / `int-quantized`). | Runtime | Supported | |---------|-----------| | **vLLM ≥ 0.21** | **Yes — preferred** (auto-detect CT; no `--quantization` flag) | | transformers + `compressed-tensors` | Yes for many text models; multimodal may need custom code | | Text Generation Inference (TGI) | **Not supported** for these CT packs | | Hugging Face Inference Widget | Often fails — use vLLM locally instead | ```bash # Preferred vllm serve 88plug/<ModelName> --trust-remote-code ``` Do **not** deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors. # Qwen3.6-27B-W8A16 INT8 post-training quantization of [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — a **vision-language model** (images + text → text) with a hybrid Gated-DeltaNet + dense-MLP architecture. **~27 GB on disk. Runs 262k token context on one A6000 or RTX 4090.** | Release tier | Gold (AutoRound iters=200) | | Quant method | AutoRound W8A16 iters=200 | | FLAC status | Not measured (T+7d milestone) | The official FP8 release requires Blackwell or H100 hardware. This quant fills the gap: near-lossless W8A16 that runs on any FP16-capable GPU — A6000, RTX 4090, L40S, A100-40. --- ## What Makes This Different ### The Gap This Fills The official Qwen3.6-27B release ships in BF16 (~54 GB) and FP8 (~28 GB). The FP8 checkpoint is great — but FP8 inference kernels require Blackwell (RTX 50xx) or Hopper (H100/H200) GPU generation. On Ampere or Ada hardware (RTX 3090/4090, A6000, A100), FP8 activations are not natively supported. This W8A16 checkpoint lands at the same ~27–28 GB footprint as the official FP8 model, but uses INT8 weights and BF16 activations — a format that Marlin kernels in vLLM serve natively on any Ampere+ GPU. No har...