--- base_model: bottlecapai/ThinkingCap-Qwen3.6-27B base_model_relation: quantized library_name: gguf tags: - qwen3_6 - gguf - llama.cpp - token-efficient - efficient-thinking pipeline_tag: image-text-to-text --- # bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF GGUF / [llama.cpp](https://github.com/ggml-org/llama.cpp) quantizations of [bottlecapai/ThinkingCap-Qwen3.6-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B) — capability of Qwen3.6-27B with **50% less** thinking tokens on average, achieved by finetuning [Qwen3.6-27B (Qwen Team, 2026)](https://huggingface.co/Qwen/Qwen3.6-27B) with online reinforcement learning while preserving the original answer quality and style. ➡️ Full model description, evaluation results (multi-seed, statistically tested), recommended sampling params, and citation: see the main model card at [bottlecapai/ThinkingCap-Qwen3.6-27B](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B). ## About GGUF and quantization [GGUF](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) is a single-file model format for running LLMs locally with llama.cpp and compatible runtimes (Ollama, LM Studio, ...). The quantized variants below store weights at reduced precision — e.g. ≈4.7 bits per weight for `Q4_K_M` instead of the 16-bit `f16` source — cutting download size and memory severalfold at a small, measured quality cost. ## Files | File | Quant | Size | |---|---|---| | `ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf` | Q4_K_M | 15.7 GB | | `ThinkingCap-Qwen3.6-27B-Q8_0.gguf` | Q8_0 | 27.1 GB | | `ThinkingCap-Qwen3.6-27B-f16.gguf` | f16 | 50.9 GB | | `mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf` | mmproj (vision) | 0.9 GB | `f16` is the unquantized source; `Q8_0` is near-lossless; `Q4_K_M` is the recommended size/quality balance for most local setups. ## Usage (llama.cpp) ```bash # pull a specific quant straight from the Hub and chat llama-cli -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M -p "Hi" # or download one file and run it huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf --local-dir . llama-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf -p "Hi" ``` ### Speculative decoding (MTP) llama.cpp can run MTP (multi-token-prediction) self-speculative decoding on these GGUFs for a decode speed-up — no separate draft model needed. Add `--spec-type draft-mtp` when serving: ```bash llama-server -hf bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF:Q4_K_M --spec-type draft-mtp ``` Set the draft length with `--spec-draft-n-max` (e.g. `4`). Requires a recent llama.cpp build with MTP support. ## Vision (image input) ThinkingCap is a **vision-language model**. Image input needs the multimodal projector `mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf` (in this repo) loaded alongside a text GGUF — the single `f16` mmproj pairs with any of the quants above. - **LM Studio / Jan / Ollama, ...:** download the `mmproj-*.gguf` from this repo; LM Studio auto-detects it and enables the image (🖼️) button. - **llama.cpp CLI:** ```bash huggingface-cli download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF \ ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --local-dir . llama-mtmd-cli -m ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf \ --mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf --image photo.jpg -p "Describe this image." ``` - **llama-server:** add `--mmproj mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf` to expose an OpenAI-compatible vision endpoint. ## Expected performance From our internal serving-validation harness (llama.cpp, single-stream, temperature 0) on a fast **N=100/dataset** subset of MMLU-Pro (reasoning) and RealWorldQA (vision) — a quick **quant-parity + decode-speed** check, *not* the headline accuracy evals (for the multi-seed, statistically-tested results see the [main model card](https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B)). Our three quants (`f16`/`Q8_0`/`Q4_K_M`) stay within subset noise of `f16` on accuracy, and MTP self-speculative decoding (`--spec-type draft-mtp`, `n=4`) accepts ≈3.75 tokens per verify step — a ≈1.4–1.7× per-token decode speed-up on top of the finetune's ≈50% token savings. `Q4_K_M` + MTP (bold) is the recommended local config. For reference we also list **[unsloth](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)'s Dynamic GGUFs of the *base* model** (`UD-*`): same llama.cpp path, but base-model quants — so they match base accuracy and reason ≈2× longer (none of the finetune's token savings). `median tokens` = median completion length; `task s` = median tokens ÷ single-stream tok/s (real per-request time); `speedup` is vs the unquantized base model in standard decoding. **MMLU-Pro (reasoning)** | config | acc | median tokens | tok/s | task s | speedup | accept_len (n=4) | |---|---|---|---|---|---|---| | Qwen3.6-27B base · standard | 0.85 | 1890 | 57.4 | 32.9 | 1.00× | — | | f16 · standard | 0.89 | 884 | 50.4 | 17.5 | 1.88× | — | | f16 · MTP | 0.88 | 870 | 86.7 | 10.0 | 3.28× | 3.78 | | Q8_0 · standard | 0.88 | 890 | 57.2 | 15.6 | 2.12× | — | | Q8_0 · MTP | 0.86 | 856 | 99.4 | 8.6 | 3.82× | 3.77 | | Q4_K_M · standard | 0.86 | 814 | 61.8 | 13.2 | 2.50× | — | | **Q4_K_M · MTP** | **0.85** | **848** | **89.2** | **9.5** | **3.46×** | **3.74** | | unsloth UD-Q8_K_XL (base) · standard | 0.85 | 1896 | 54.5 | 34.8 | 0.95× | — | | unsloth UD-Q8_K_XL (base) · MTP | 0.86 | 1925 | 98.2 | 19.6 | 1.68× | 3.74 | | unsloth UD-Q4_K_XL (base) · standard | 0.84 | 1976 | 62.1 | 31.8 | 1.03× | — | | unsloth UD-Q4_K_XL (base) · MTP | 0.83 | 1928 | 87.1 | 22.1 | 1.49× | 3.72 | **RealWorldQA (vision)** | config | acc | median tokens | tok/s | task s | speedup | accept_len (n=4) | |---|---|---|---|---|---|---| | Qwen3.6-27B base · standard | 0.74 | 556 | 57.4 | 9.7 | 1.00× | — | | f16 · standard | 0.79 | 271 | 50.4 | 5.4 | 1.80× | — | | f16 · MTP | 0.79 | 271 | 86.7 | 3.1 | 3.10× | 3.78 | | Q8_0 · standard | 0.79 | 270 | 57.2 | 4.7 | 2.05× | — | | Q8_0 · MTP | 0.78 | 273 | 99.4 | 2.7 | 3.53× | 3.77 | | Q4_K_M · standard | 0.78 | 283 | 61.8 | 4.6 | 2.11× | — | | **Q4_K_M · MTP** | **0.78** | **274** | **89.2** | **3.1** | **3.15×** | **3.74** | | unsloth UD-Q8_K_XL (base) · standard | 0.68 | 530 | 54.5 | 9.7 | 1.00× | — | | unsloth UD-Q8_K_XL (base) · MTP | 0.69 | 550 | 98.2 | 5.6 | 1.73× | 3.74 | | unsloth UD-Q4_K_XL (base) · standard | 0.65 | 655 | 62.1 | 10.5 | 0.92× | — | | unsloth UD-Q4_K_XL (base) · MTP | 0.70 | 564 | 87.1 | 6.5 | 1.49× | 3.72 |
