--- license: apache-2.0 base_model: tencent/Hy3 base_model_relation: quantized pipeline_tag: text-generation quantized_by: vcruz305 tags: - gguf - imatrix - moe - hy3 - tencent - conversational --- # Hy3-GGUF imatrix GGUF quantizations of [tencent/Hy3](https://huggingface.co/tencent/Hy3) — **295B total / 21B active** MoE (192 experts, top-8), 80 layers + 1 MTP/NextN layer (3.8B), 256K context. Quantized day-zero from the BF16 release and **smoke-tested on real hardware before upload** (NVIDIA DGX Spark, GB10). Performance numbers below are measured, not estimated. ## ⚠️ Requires llama.cpp PR #25364 (unmerged) The `hy_v3` architecture is not yet in llama.cpp master. Until [PR #25364](https://github.com/ggml-org/llama.cpp/pull/25364) merges, build from the PR branch: ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp git fetch origin pull/25364/head:hy3-port && git checkout hy3-port cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release cmake --build build --config Release -j"$(nproc)" ``` These quants were produced at PR head `a4da4b5cfdc4e5fa9def068e216a6e5154f22848`. ## Quants All quants use an importance matrix (`Hy3.imatrix`, included) computed on a ~63KB diverse coding/reasoning/chat calibration corpus. Files are sharded at ~48GB for HF's 50GB limit — download the whole folder and point llama.cpp at the `-00001-of-*` shard; the rest load automatically. | Quant | Size | ~BPW | Fits | Notes | |---|---|---|---|---| | [Q8_0](https://huggingface.co/vcruz305/Hy3-GGUF/tree/main/Hy3-Q8_0) | 318 GB | 8.6 | server-class | near-lossless | | [Q5_K_M](https://huggingface.co/vcruz305/Hy3-GGUF/tree/main/Hy3-Q5_K_M) | 212 GB | 5.8 | 2× 128GB-class | | | [Q4_K_M](https://huggingface.co/vcruz305/Hy3-GGUF/tree/main/Hy3-Q4_K_M) | 181 GB | 4.9 | 2× 128GB-class | recommended dual-node; verified over RPC | | [IQ4_XS](https://huggingface.co/vcruz305/Hy3-GGUF/tree/main/Hy3-IQ4_XS) | 159 GB | 4.3 | 2× 128GB-class | | | [Q3_K_M](https://huggingface.co/vcruz305/Hy3-GGUF/tree/main/Hy3-Q3_K_M) | 143 GB | 3.9 | 2× 128GB-class | | | [IQ3_XXS](https://huggingface.co/vcruz305/Hy3-GGUF/tree/main/Hy3-IQ3_XXS) | 117 GB | 3.2 | single 128GB-class (borderline) | MTP block @ q8_0; best quality-per-GB single-box | | [Q2_K](https://huggingface.co/vcruz305/Hy3-GGUF/tree/main/Hy3-Q2_K) | 109 GB | 3.0 | single 128GB-class | | | [IQ2_M](https://huggingface.co/vcruz305/Hy3-GGUF/tree/main/Hy3-IQ2_M) | 100 GB | 2.7 | single 128GB-class | MTP block @ q8_0; smallest tier | On low-bit IQ tiers the MTP/NextN layer (`blk.80.*`) is kept at q8_0 (`--tensor-type`) — very-low-bit quantization of that block is not possible without imatrix coverage, and this preserves it intact for future speculative decoding support. ## Measured performance (DGX Spark, GB10, 273GB/s unified memory) Single node, fully GPU-resident, `-fa on`: | Quant | Gen tok/s | + MTP spec decode | Prompt tok/s | |---|---|---|---| | IQ2_M | 18.0–18.5 | **22.8 (+27%)** | 36–50 | | Q2_K | 18.1 | untested | 35.5 | | IQ3_XXS | 17.1 | untested | 32.4 | MTP numbers measured at temp 0 with `--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75` (90% draft acceptance). Higher sampling temperatures reduce acceptance and land between the two columns. Dual node (Q4_K_M, 181GB layer-split across 2× GB10 over 200GbE via llama.cpp RPC): **14.0 tok/s gen / 21.9 tok/s prompt**. RPC layer-split adds capacity for bigger quants, not speed — expect single-node-or-slower decode rates. All smoke-tested tiers (IQ2_M, Q2_K, IQ3_XXS single-node; Q4_K_M dual-node) produced coherent output (code generation + chat), with the Hy3 chat template engaging correctly via `--jinja`. The remaining tiers (Q8_0, Q5_K_M, IQ4_XS, Q3_K_M) were produced by the same verified pipeline but not individually load-tested — report any issues in the community tab. ## Running ```bash # chat/completion (--jinja is required — the Hy3 template is not natively supported) ./build/bin/llama-completion \ -m Hy3-Q2_K/Hy3-Q2_K-00001-of-00003.gguf \ -ngl 99 -fa on -c 8192 --jinja \ -p "Write a Python function that returns the median of a list." -n 256 # server ./build/bin/llama-server \ -m Hy3-Q2_K/Hy3-Q2_K-00001-of-00003.gguf \ -ngl 99 -fa on -c 8192 --jinja --host 0.0.0.0 --port 8080 ``` Tips: - First load of a 100GB+ quant can take several minutes — don't kill it early. - Constrained on memory? Offload MoE expert tensors to CPU with `--n-cpu-moe N`. - `--jinja` matters: without it the chat template aborts on current llama.cpp. ## MTP / speculative decoding status **Verified working (2026-07-07):** [PR #25395](https://github.com/ggml-org/llama.cpp/pull/25395) adds Hy3 MTP speculative decoding, and these GGUFs work with it as-is — the MTP tensors bundled in every quant (q8_0-preserved on low-bit tiers) are used directly as the `draft-mtp` target. Measured on IQ2_M: **18.0 → 22.8 tok/s (+27%), 90% draft acceptance**. ```bash ./build/bin/llama-server -m Hy3-IQ2_M/Hy3-IQ2_M-00001-of-00003.gguf \ -ngl 99 -fa on -c 32768 --jinja \ --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75 \ --parallel 1 ``` Notes: - `--spec-draft-p-min 0.75` is **required for a speedup** — the MTP head is trained single-depth, and the default p_min makes speculation a net loss (per the PR author's measurements, confirmed here). - `--parallel 1` is required for draft-mtp; `n_max` 2 and 3 measure within ~1% of each other (n=2 slightly ahead at 90% acceptance). - `--spec-type` exists on `llama-server` and `llama-cli` only, not `llama-completion`. ### ⚠️ If you downloaded before 2026-07-08: arch string fix PR #25395 renamed the architecture string from `hy-v3` (the earlier PR #25364) to `hy_v3`. All first-shards in this repo were re-uploaded with the fix on 2026-07-07, so fresh downloads just work. If you hold older files and see `unknown model architecture: 'hy-v3'`, either re-download the first shard of your quant, or patch in place (the string lives only in shard 00001's header; byte-for-byte same length): ```python # python patch_arch.py — swaps hy-v3 -> hy_v3 in the header import mmap, sys with open(sys.argv[1], "r+b") as f: mm = mmap.mmap(f.fileno(), 64 * 1024 * 1024) # metadata lives well within 64MB i = mm.find(b"hy-v3") while i != -1: mm[i:i+5] = b"hy_v3"; i = mm.find(b"hy-v3", i + 1) mm.flush() ``` ## Provenance - Source: [tencent/Hy3](https://huggingface.co/tencent/Hy3) (BF16, 597.6GB, 99 shards) - llama.cpp: [PR #25364](https://github.com/ggml-org/llama.cpp/pull/25364) @ `a4da4b5cfdc4e5fa9def068e216a6e5154f22848` - imatrix computed on a Q8_0 intermediate (standard practice for models whose BF16 GGUF exceeds node RAM) - Quantized and validated on a 2-node NVIDIA DGX Spark cluster Quantized by [vcruz305](https://huggingface.co/vcruz305). Please report issues in the community tab.
