Step 3.7 Flash Quark W8A8

--- base_model: Step-3.7-Flash pipeline_tag: image-text-to-text library_name: transformers tags: - quantization - compressed-tensors - vllm - w8a8 - moe --- # Step-3.7-Flash W8A8 (AMD Quark → compressed-tensors) Symmetric **INT8** weights (per-channel) with **INT8 dynamic per-token** activations. - **Base model**: Step-3.7-Flash (multimodal `Step3p7ForConditionalGeneration`, 196B total / A11B sparse MoE, 288 routed experts top-8) - **Quantization tool**: AMD Quark 0.11.2 (`int8`, file2file), then losslessly repacked to the **compressed-tensors** format so that vLLM can load it. - **Calibration**: weight-only PTQ (RTN), no activation calibration. Activations are quantized dynamically per-token at inference time (no calibration needed). - **Excluded from quantization**: `lm_head`, MoE router (`moe.gate`, `router_bias`), `share_expert`, `self_attn.g_proj`, dense layers 0-2, MTP layers 45-47, and the full vision tower. ## Usage (vLLM) ```bash vllm serve --trust-remote-code --tensor-parallel-size 2 --enable-expert-parallel ``` > NOTE: vLLM gates INT8 MoE behind `current_platform.is_cuda()`, which excludes ROCm. The Triton INT8 MoE kernel actually runs fine on MI355X; patch `fused_moe/experts/triton_moe.py` to force `device_supports_int8 = True` before launching.