Qwen3.6 27B NVIDIA NVFP4 MTP Q8attn GGUF

GGUF conversion of nvidia/Qwen3.6-27B-NVFP4, preserving NVIDIA's NVFP4 MLP tensors, with MTP speculative decoding and a BF16 vision projector.

This is the accuracy-oriented variant. Compared to the smaller Q4-attention build, this one keeps the attention and linear-attention (DeltaNet) projections at Q8_0 — matching the original FP8 precision of the NVIDIA checkpoint — instead of dropping them to Q4_K. That adds about 3 GB to the file size, with the aim of reducing accuracy loss at long context, where 4-bit attention / DeltaNet projections hurt most. Benchmarked on an RTX 5090 with llama-benchy.

Highlights

• NVFP4 preserved: 193 NVFP4 MLP tensors are kept from NVIDIA's ModelOpt quantized checkpoint (unchanged from the Q4-attention build).
• Q8 attention: attention (q/k/v/o) and linear-attention / DeltaNet projections (attn_qkv, attn_gate, ssm_out, ssm_alpha, ssm_beta) are stored as Q8_0 (≈ the original FP8), for better accuracy than the Q4-attention build.
• MTP included: the GGUF keeps the extra MTP layer for draft-mtp speculative decoding.
• Vision supported: includes the same BF16 mmproj file as the Q4-attention build for image input.
• RTX 5090 tested: measured with llama-benchy using MTP depth d=3.

Which build should I pick?

Build	Attention / DeltaNet precision	Size	Pick it for
Q4-attention (sibling)	Q4_K	14.66 GiB	Smallest footprint
Q8attn (this repo)	Q8_0	17.81 GiB	Better accuracy, esp. long context

Both builds share the same 193 NVFP4 MLP tensors, MTP layer, and BF16 vision projector. Generation speed with MTP d=3 is comparable (~110–130 tok/s on RTX 5090), so the choice is accuracy vs. size, not throughput.

Provenance

Component	Source
Base model	Qwen/Qwen3.6-27B
NVFP4 source checkpoint	nvidia/Qwen3.6-27B-NVFP4
Runtime target	llama.cpp

Files

File	Size	Description
Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn.gguf	17.81 GiB / 19,128,366,464 bytes	Main GGUF. NVFP4 MLP preserved; attention/DeltaNet at Q8_0; MTP layer included.
mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf	~888 MiB / 931,146,304 bytes	BF16 vision projector for image input.

llama.cpp example

# Text-only
llama-server \
  -m Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -c 196608 \
  -ngl 999

# With vision
llama-server \
  -m Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn.gguf \
  --mmproj mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  -c 147456 \
  -ngl 999

The context lengths above are what this repo's author runs on a 32 GB RTX 5090: -c 196608 for text-only and -c 147456 when the vision projector is loaded (the mmproj plus its image tokens take extra VRAM, so a smaller context leaves headroom). Adjust to fit your own VRAM.

Benchmarks

Measured on RTX 5090, llama-benchy, MTP depth d=3, ctx 32768, generation-latency mode. (Absolute tok/s is not 1:1 comparable to the sibling card's table: this run used a different llama.cpp build and a gpt2 tokenizer fallback. Use it to compare thinking on/off within this table.)

Mode	Prompt / gen length	Generation (tg) tok/s	Peak gen tok/s
thinking ON	pp512 / tg512	125.28 ± 2.67	~135
thinking ON	pp4096 / tg512	127.05 ± 0.96	~135
thinking OFF	pp512 / tg512	115.35 ± 10.30	—
thinking OFF	pp4096 / tg512	110.04 ± 9.12	—

MTP d=3 roughly doubles generation throughput over the non-speculative baseline on this hardware.

Model details

Field	Value
GGUF architecture	qwen35
License metadata	apache-2.0
Text layers	64
MTP layers	1
qwen35.block_count	65
qwen35.nextn_predict_layers	1
Bits per weight	~5.60 BPW
Vision projector architecture	clip / qwen3vl_merger
Vision projector precision	BF16/F32

Tensor composition of the main GGUF

Tensor type	Count	Role
NVFP4	193	MLP (gate/up/down) — preserved from the NVIDIA NVFP4 checkpoint
Q8_0	308	attention (q/k/v/o) + linear-attn / DeltaNet projections
Q6_K	3	MTP-layer FFN
Q4_K	2	remaining small tensors
F32	746	norms / embeddings / scales
Total	1252

The conversion keeps the original NVFP4 MLP tensors instead of requantizing all weights to K-quants, and, unlike the Q4-attention build, keeps attention / DeltaNet projections at Q8_0 rather than Q4_K.

License and attribution

The upstream models are licensed under Apache-2.0:

• Qwen/Qwen3.6-27B
• nvidia/Qwen3.6-27B-NVFP4

This repository redistributes a GGUF format conversion of the NVIDIA NVFP4 checkpoint under the same Apache-2.0 license. Please also follow the terms and usage guidance of the upstream model cards.

Acknowledgements

Thanks to the Qwen team for the base model, NVIDIA for the NVFP4 ModelOpt checkpoint, and the llama.cpp project for GGUF, NVFP4, MTP, and multimodal runtime support.