--- license: mit base_model: zai-org/GLM-5.2 base_model_relation: quantized library_name: transformers pipeline_tag: text-generation tags: - glm - moe - nvfp4 - aqlm - quantized - 1m-context language: - en - zh --- # GLM-5.2 NVFP4+AQLM hybrid — 1M context on 4x RTX PRO 6000 (SM120, 384 GB) A 744B-param **GLM-5.2** quantized with a **per-expert two-tier** scheme so it serves a **full ~1M-token KV cache on 4x 96 GB SM120 GPUs**. This build adds, over the first release: a **measured-error expert re-tier** (byte-neutral hot/cold swaps moving NVFP4 budget out of early layers, where 2-bit reconstruction is nearly free, into deep layers where it was starving) and **PV-tuning of all cold codebooks/scales** against the BF16 teacher under the new assignment (held-out early stopping). - **Hot experts — NVFP4 @ 4.5 bpw**: the ~30% hottest experts per layer (65–96 of 256), ranked by routing mass over a 1.04M-token coding / tool-calling / medical calibration corpus. They cover ~50% of routed computations. - **Cold experts — AQLM @ 2 bpw**: the rest (1x16 codebooks over groups of 8, per-expert per-channel fp16 scales). - Attention, DSA indexers, dense MLPs, shared experts, embeddings: **BF16**. - KV cache: **fp8_ds_mla** (656 B/token/layer packed MLA) — required; it's what lets the long context fit. Cold-expert codebooks and scales are **PV-tuned** against the BF16 teacher's MoE outputs (per-layer output-MSE objective, held-out early stopping) with the hot experts' NVFP4 contribution subtracted, so the 2-bit tier compensates for this build's specific hot/cold assignment. Weights: **292 GB**, sharded across 4 GPUs. **Validated on SM120 (4x RTX PRO 6000):** coherent generation, needle-in-haystack retrieval **PASS at 134K tokens** under the recommended TP4+DCP4 serving config, full ~1M-token KV pool fits. ## Quality (teacher-forced perplexity, same harness for all models) | eval set | this repo | [REAP-504B-v2](https://huggingface.co/0xSero/glm-5.2-reap-504B-v2) (168/256 experts, 308 GB) | [GLM-5.2-504B-Nvidia](https://huggingface.co/0xSero/GLM-5.2-504B-Nvidia) (NVFP4 REAP, 318 GB) | |---|---|---|---| | code+docs+medical, 159k tok | **2.5223** | 2.6961 | 2.8784 | | wikitext-2 slice | **3.7024** | 6.8941 | 9.0968 | | fresh GitHub (post-2026-06) | 2.2910 | **2.2603** | 2.3460 | | code+docs, 46.7k tok | 2.7393 | **2.7347** | 2.8661 | All 256 experts per layer remain routable (nothing pruned); the REAP comparisons drop 88 experts/layer → large regressions on prose and medical/general-knowledge content. Sibling variants trading context for more NVFP4: [500k / 48% hot](https://huggingface.co/jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-500k) · [250k / 57% hot](https://huggingface.co/jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-250k) --- # Serving (4x RTX PRO 6000, SM120 — TP4 + DCP4) ## 1. The vLLM fork Not stock vLLM. Use the pre-patched fork (no runtime patching): > **github.com/jarrelscy/vllm-glm52-sm120** (branch **`glm52-sm120`**) It builds on a recent upstream vLLM that already ships **DCP** (`--decode-context-parallel-size`, decode context parallelism for MLA models, vLLM PR #46076) and native **MTP** speculative decode; the fork adds the hybrid **NVFP4+AQLM** MoE loader/kernels, the **SM120 sparse-MLA (DSA) fixes** that make DCP and MTP coherent on RTX PRO 6000, and PIECEWISE-graphed draft decodes. ```bash git clone -b glm52-sm120 https://github.com/jarrelscy/vllm-glm52-sm120 cd vllm-glm52-sm120 && docker build -f Dockerfile.glm52-sm120 -t glm52-sm120 . ``` ## 2. Recommended (and default) config — `tp4-1m-mtp` **TP4 + DCP4 + native MTP (ns=3) + PIECEWISE CUDA graphs + tool calling.** Coherent ~1M window at TP speed with lossless MTP self-speculation. This is the default. ```bash docker run --gpus all --ipc=host -p 8001:8001 \ -v /path/to/weights:/models/1m:ro \ -e PARALLEL=tp4-1m-mtp glm52-sm120 ``` Equivalent bare launch: ```bash export NCCL_MAX_NCHANNELS=4 NCCL_BUFFSIZE=1048576 VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=192 vllm serve \ --tensor-parallel-size 4 \ --decode-context-parallel-size 4 --dcp-comm-backend a2a \ --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":3}' \ --compilation-config '{"mode":3,"cudagraph_mode":"PIECEWISE"}' \ --gpu-memory-utilization 0.95 --kv-cache-dtype fp8_ds_mla \ --max-model-len 950000 --max-num-seqs 2 --max-num-batched-tokens 2048 \ --no-enable-flashinfer-autotune \ --enable-auto-tool-choice --tool-call-parser glm47 \ --served-model-name glm-5.2 --port 8001 ``` - **`cudagraph_mode` MUST be `PIECEWISE`** for DCP+spec (`FULL` deadlocks). Do not add `--enforce-eager`. - **ns=3 is the tuned optimum**: single-stream verify batch = ns+1 = 4 hits the exact graph-4 replay; ns≥4 pads verify to graph-8 (2× work). Override with `-e NUM_SPEC=`. - **Tool/function calling** is on (`glm47` parser); the chat template ships ` ` tags. `-e ENABLE_TOOLS=0` to disable. - The DCP output-combine uses the `a2a` (all-to-all) backend (a single all_to_all + local combine, ~5% faster decode than ag_rs on this PCIe box; `DCP_BACKEND=ag_rs` to revert). DCP shards the MLA latent KV by sequence across the 4 ranks (plain TP4 replicates KV and caps out early); MTP's MLA-shaped draft KV shares that sharded latent, so ~1M + lossless spec co-fit. ## 3. Alternatives (set `-e PARALLEL=`) | config | what | window / speed | |--------|------|----------------| | **tp4-1m-mtp** ★ default | TP4+DCP4+MTP ns3 + graphs + tools | ~1M + lossless spec | | tp4-1m | TP4+DCP4, no spec | ~1M — **best multi-stream throughput** | | tp4-dspark | TP4 + DSpark | ~247K — fastest short-context | --- # Performance (measured, 4x RTX PRO 6000 SM120) - **Single-stream decode (`tp4-1m-mtp`, ns=3):** ~**57 tok/s @32K** (counting & essay), lossless. - **Prefill:** ~720 tok/s @32K · ~970 @128K · ~1,440 @256K (prefix-cached across requests). - **Concurrency / throughput @32K** (aggregate decode tok/s): MTP saturates ~64; **no-spec `tp4-1m` scales to ~128 (~2×)** — spec is a single-user *latency* play, no-spec is the multi-user *throughput* play. Use `tp4-1m` + raise `--max-num-seqs` for serving many streams. > Measure spec decode with **non-streamed** responses — under MTP, vLLM bundles multiple accepted tokens into one SSE delta, so chunk-counting undercounts by ~the acceptance factor. `code/` contains the production pipeline (routing stats, per-expert assignment solver, checkpoint builders) and `SETUP.md`. Quantized from [lukealonso/GLM-5.2-NVFP4](https://huggingface.co/lukealonso/GLM-5.2-NVFP4).
