--- base_model: - prefeitura-rio/Rio-3.5-Open-397B license: mit tags: - qwen3.5 - moe - quantized - nvfp4 - fp4 - multimodal pipeline_tag: text-generation --- # Rio-3.5-Open-397B-NVFP4 NVFP4-quantized build of [prefeitura-rio/Rio-3.5-Open-397B](https://huggingface.co/prefeitura-rio/Rio-3.5-Open-397B) — a 397B-parameter (17B active) Qwen3.5-MoE vision-language model (512 experts, hybrid softmax + linear/DeltaNet attention), itself a finetune of Qwen/Qwen3.5-397B-A17B. edit: ir daid ir was rhese thinfs. but it appears to be. nex n2 with a system prompt laugh out loud Quantized with [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.44 using the per-expert streaming calibration pipeline from [local-inference-lab/quant-toolkit](https://github.com/local-inference-lab/quant-toolkit) (Luke Alonso). Produced on 8x B200. Size on disk: ~251 GB (46 shards). ## What's quantized Only the **routed** MoE expert MLPs (gate/up/down) are NVFP4 (4-bit, blockwise FP8 scales, group size 16), calibrated **per-expert**. Left in BF16: shared-expert MLPs (active every token), attention (softmax + DeltaNet), router/gates, vision tower, MTP, embeddings, lm_head. KV cache is FP8 (e4m3). This mirrors [lukealonso/Qwen3.5-397B-A17B-NVFP4](https://huggingface.co/lukealonso/Qwen3.5-397B-A17B-NVFP4). - quant_algo: `NVFP4` | quant_method: `modelopt` - kv_cache_scheme: `{'dynamic': False, 'num_bits': 8, 'type': 'float'}` ## Calibration Per-expert max-calibration over a finetune-appropriate subset (Rio tracks the Qwen3.5 base, so the full base-model corpus is unnecessary): deep-reasoning + diverse-instruction + agentic-coding corpora (the same datasets as Luke's Qwen3.5 recipe). Post-calibration: rare-expert amaxes floored to median/10; gate/up weight amaxes tied for fused w13 export. Master amaxes published under `calibration/` for reproducible re-export. ## How to run (SGLang) ```bash python3 -m sglang.launch_server \ --model brandonmusic/Rio-3.5-Open-397B-NVFP4 \ --served-model-name Rio-3.5 \ --reasoning-parser qwen3 --tool-call-parser qwen3_coder \ --tensor-parallel-size 4 \ --quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 \ --trust-remote-code \ --speculative-algo NEXTN --speculative-num-steps 5 \ --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 \ --mamba-scheduler-strategy extra_buffer \ --mem-fraction-static 0.9 --host 0.0.0.0 --port 8000 ``` ## Acknowledgements - **Luke Alonso** — the per-expert NVFP4 [quant-toolkit](https://github.com/local-inference-lab/quant-toolkit) this build uses. - **prefeitura-rio** — the base model, Rio-3.5-Open-397B. - **Qwen** — Qwen3.5-397B-A17B, the underlying architecture.
