Qwen3.6 35B A3B W8A16

--- license: apache-2.0 base_model: Qwen/Qwen3.6-35B-A3B base_model_relation: quantized language: - en tags: - qwen3 - qwen - moe - mixture-of-experts - vision - image-text-to-text - multimodal - quantized - int8 - INT8 - w8a16 - compressed-tensors - vllm - autoround - llmcompressor - long-context - 1m-context - gated-deltanet - linear-attention - 8-bit - ptq - safetensors - sglang - 88plug - post-training-quantization - thinking - reasoning pipeline_tag: image-text-to-text library_name: transformers model_type: qwen3_5_moe --- ## Load path (important) These weights are **compressed-tensors** (`pack-quantized` / `int-quantized`). | Runtime | Supported | |---------|-----------| | **vLLM ≥ 0.21** | **Yes — preferred** (auto-detect CT; no `--quantization` flag) | | transformers + `compressed-tensors` | Yes for many text models; multimodal may need custom code | | Text Generation Inference (TGI) | **Not supported** for these CT packs | | Hugging Face Inference Widget | Often fails — use vLLM locally instead | ```bash # Preferred vllm serve 88plug/<ModelName> --trust-remote-code ``` Do **not** deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors. # Qwen3.6-35B-A3B-W8A16 INT8 post-training quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — a **vision-language model** (images + video + text → text) with a hybrid Gated-DeltaNet + sparse-MoE architecture. **35 GB on disk. Runs 1M token context on one A100/H100.** | Release tier | Gold (AutoRound iters=200) | | Quant method | AutoRound W8A16 iters=200 | | FLAC status | Not measured (T+7d milestone)| Most published Qwen3.6 quants silently degrade because they skip the tail-expert calibration starvation problem. This quant addresses it directly. --- ## What Makes This Different ### The Problems with Typical MoE Quants Published quantizations of Qwen3.6-35B-A3B tend to fail in two specific ways: **1. Long-context Q/K outliers.** Rotary embeddings produce large-magnitude outliers in Q and K projections that standard quantization handles poorly. The effect is subtle at short context but compounds significantly beyond 32k tokens — exactly where this model's KV-light architecture is most valuable. **2. Tail-expert calibration starvation.** With 256 experts per MoE layer and only 8 routed per token...