Step 3.7 Flash Quark W4A16

--- base_model: Step-3.7-Flash pipeline_tag: image-text-to-text library_name: transformers tags: - quantization - compressed-tensors - vllm - w4a16 - moe --- # Step-3.7-Flash W4A16 (AMD Quark → compressed-tensors) Symmetric **INT4** weight-only quantization (group_size=128). - **Base model**: Step-3.7-Flash (multimodal `Step3p7ForConditionalGeneration`, 196B total / A11B sparse MoE, 288 routed experts top-8) - **Quantization tool**: AMD Quark 0.11.2 (`int4_wo_128`, file2file), then losslessly repacked to the **compressed-tensors** format so that vLLM can load it. - **Calibration**: weight-only PTQ (RTN), no activation calibration. Activations stay in BF16. - **Excluded from quantization**: `lm_head`, MoE router (`moe.gate`, `router_bias`), `share_expert`, `self_attn.g_proj`, dense layers 0-2, MTP layers 45-47, and the full vision tower. ## Usage (vLLM) ```bash vllm serve --trust-remote-code --tensor-parallel-size 2 --enable-expert-parallel ``` Loads natively in vLLM (CompressedTensors WNA16 MoE).