--- license: apache-2.0 language: - en base_model: empero-ai/Qwable-9B-Claude-Fable-5 pipeline_tag: text-generation library_name: gguf tags: - qwen - qwen3.5 - 9b - quantized - quantization - gguf - llama-cpp - q5_k_m - q8_0 - q6_k - imatrix - hybrid-quantization - selective-quantization - shq - SHQ8 - lightweight - coding - code-generation - llm - open-source - empero - claude-fable-5 - deltanet - gated-attention --- # Qwable-9B-Claude-Fable-5 — SHQ Family (Selective Hybrid Quants) > Adaptive quantization family for Qwen3.5-9B hybrid Gated DeltaNet + Full Attention architecture. Each variant uses real imatrix data to selectively assign precision per-tensor. **Model:** [empero-ai/Qwable-9B-Claude-Fable-5](https://huggingface.co/empero-ai/Qwable-9B-Claude-Fable-5) — Claude Fable 5 reasoning fine-tune of Qwen3.5-9B. > **Note:** The model file `SHQ8-Q5_K_M.gguf` is named for HF parser compatibility. This is **NOT a pure Q5_K_M quantization** — it's a hybrid using Q5_K_M base with Q8_0, Q5_K, Q4_K, and F16 selectively applied across different tensor types. Check the config for exact per-tensor types. > **Note:** `SHQ8-v2-Q5_K_M.gguf` is also a hybrid — Q5_K_M base with Q8_0 on critical attention, Q6_K on mid attention, IQ4_XS on low-importance tensors. See `configs/SHQ8_v2.sh` for the exact per-tensor map. **Speed:** ~22 tokens/sec on GTX 1070 (8 GB VRAM) with flash-attn for SHQ-OptA, **~26 tokens/sec for SHQ8-v2** (fewer Q8_0 tensors → less data through Pascal's PCIe). > I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback. ## SHQ Family Tree ``` SHQ Series (Selective Hybrid Quantization) │ ├── SHQ6 ── Gemma 4 12B IT QAT │ (QAT model, IQ4_XS all FFN + Q8_0 attention) │ └── gemma4/README.md │ ├── SHQ7 ── Gemma 4 12B OBLITERATED │ (non-QAT, gradient IQ4_NL + Q4_K + IQ4_XS + Q8_0) │ └── gemma4/OBLITERATED_SHQ7_README.md │ ├── SHQ8 ── Qwen3.5-9B fine-tunes (non-QAT, Gated DeltaNet hybrid) │ │ │ ├── Qwable-9B-Claude-Fable-5 (this) │ │ ├── SHQ-OptA ★ WINNER │ │ │ Q5_K_M base + Q8_0 on attn_gate/qkv/ssm_alpha/beta + imatrix │ │ │ └── 6303 MiB, PPL 7.3747 — beats Q6_K in size + quality │ │ ├── SHQ8-v2 — compact champion │ │ │ Q5_K_M + selective Q8_0/Q6_K/IQ4_XS by imatrix │ │ │ └── 5726 MiB, PPL 7.4559 — 18% smaller than OptA │ │ ├── SHQ8-v3 — IQ4_NL experiment │ │ │ Q5_K_M + all attn Q8_0 + IQ4_NL on imp ffn_down │ │ │ └── 6041 MiB, PPL 7.4456 — 4% smaller than OptA │ │ ├── SHQ8-v3a — all-attn Q8_0 test │ │ │ Q5_K_M + all attn Q8_0 + ffn_down blk.20-30 Q8_0 │ │ │ └── 6301 MiB, PPL 7.4310 — same size, +0.056 vs OptA │ │ └── Exp1 ── aborted │ │ IQ4_XS all FFN (wrong for non-QAT) │ │ │ └── Qwythos-9B-Claude-Mythos-5-1M │ ├── SHQ8-OptA ★ quality champion │ │ Same formula as OptA above │ │ └── 6303 MiB, PPL 7.4831 — best PPL on Qwythos │ ├── SHQ8-v2 — compact champion │ │ Same tiered IMatrix approach as v2 above │ │ └── 5726 MiB, PPL 7.6542 — 577 MiB less than OptA │ └── MTP variants (built-in draft head for speculative decoding) │ ├── M-SHQ8-MTP-OptA │ │ OptA + blk.32 (MTP) at Q8_0 │ │ └── 6.40 GB, PPL 7.4831 — quality champion │ └── M-SHQ8-MTP-v2 │ v2 + blk.32 (MTP) at Q8_0 │ └── 5.83 GB, PPL 7.6542 — compact champion ``` ## TL;DR | Quant | Size | PPL (ctx=1024) | vs Q6_K | vs OptA | |---|---|---|---|---| | **i1-Q6_K (baseline)** | 7008 MiB | 7.4394 ± 0.05 | — | — | | **SHQ-OptA** ★ quality | 6303 MiB | **7.3747 ± 0.05** | **best PPL** | — | | **SHQ8-v2** ★ compact | 5726 MiB | 7.4559 ± 0.05 | +0.017, -1282 MiB | +0.081, -577 MiB | | SHQ8-v3 | 6041 MiB | 7.4456 ± 0.05 | +0.006, -967 MiB | +0.071, -262 MiB | | SHQ8-v3a | 6301 MiB | 7.4310 ± 0.05 | -0.008, -707 MiB | +0.056, -2 MiB | **SHQ-OptA** is the quality champion (best PPL). **SHQ8-v2** is the compact champion (18% smaller than OptA at +0.081 PPL). ## Key Result **Q5_K_M base + Q8_0 on attn_gate/qkv/ssm_alpha/ssm_beta + blk.31.ffn_down Q8_0 + imatrix per-row refinement.** - **PPL 7.3747** vs Q6_K 7.4394 (Δ = -0.065, within margin but directionally better) - **Size 6303 MiB** vs Q6_K 7008 MiB (10% smaller) - Imatrix auto-promotes high-importance tensors (ffn_down in deep layers → Q6_K) - Manual overrides take precedence: attn_gate/qkv → Q8_0 everywhere, blk.31.ffn_down → Q8_0 ## SHQ8-v2 — Real IMatrix Tiered Precision **Q5_K_M base + imatrix per-row + tiered precision by real importance data.** Real importance values from the actual `Qwable-9B-Claude-Fable-5.imatrix.gguf` (248 tensors, 319 chunks) revealed a **33,000× range** — from blk.31 ffn_down at 323K to blk.0-5 ssm_out at 15. This enables safe aggressive quantization on low-importance tensors. | Tier | Types | Size Impact | Rationale | |------|-------|-------------|-----------| | **Q8_0** | blk.0 attn (L0), blk.26-31 attn, blk.31 ffn_down, blk.31 attn_output, norms | +~155 MiB vs Q5_K | Highest importance layer edges | | **Q6_K** | blk.1-25 attn_gate/qkv/ssm_alpha/beta | -~170 MiB vs Q8_0 | Mid layers don't need Q8_0 | | **IQ4_XS** | ssm_out (all), attn_output (all), ffn_down (except blk.31) | -~350 MiB vs Q5_K | Low importance (43-21K mean) | | **F16** | ssm_dt, ssm_a, all norms | Negligible | Tiny tensors | | **Q5_K_M** | Everything else + imatrix per-row | Base | Proven design | **Key insight — real imatrix ≠ README claims:** The README's importance matrix table (1M+ for attention) was from a different file. The actual model's imatrix shows ~2× lower values and a different layer ordering (L0 dominates at 494K, not mid layers). This changes which tensors need high precision. **Size:** 5,726 MiB (5.36 BPW) — **577 MiB (-9.2%) less than SHQ-OptA**. **PPL:** 7.4559 ± 0.05 — matches Q6_K (7.4394) within margin at 18% smaller size. Slightly higher than SHQ-OptA (7.3747) but saves 577 MiB (9.2%). A favorable size/quality trade-off. **Speed:** ~26 tokens/sec on GTX 1070 — 18% faster than SHQ-OptA thanks to fewer Q8_0 tensors reducing VRAM bandwidth pressure. ## Architecture Same as Qwen3.5-9B — hybrid Gated DeltaNet + Gated Attention (3:1 pattern): | Property | Value | |---|---| | Layers | 32 (24 DeltaNet + 8 Full Attention) | | Hidden dim | 4096 | | FFN intermediate | 12288 | | Vocabulary | 248,320 | | Attention-only | blk.3, 7, 11, 15, 19, 23, 27, 31 | | DeltaNet layers | all others | Full architecture details in [SHQ-QWEN35-INSIGHTS_CODER.md](SHQ-QWEN35-INSIGHTS_CODER.md). ## How We Made It ### Step 1: Q6_K baseline (PPL 7.4394) Standard Q6_K with mradermacher's imatrix. Fits 8 GB VRAM at 7008 MiB. ### Step 2: The IQ4_XS trap (aborted) We tried IQ4_XS on all FFN tensors across all layers — `blk.\d+.ffn_(gate|up|down)=IQ4_XS`. This was a **bad design for a non-QAT model**: ffn_gate/up have 572K importance (far above the 150K IQ4_XS safety threshold). The run was aborted before completion. ### Step 3: CODER insights rescue Existing experiments on Qwopus3.5-9B-Coder-MTP (same architecture, different fine-tune) provided critical evidence (tested at ctx=2048): | Experiment | Result | Lesson | |---|---|---| | SHQv2 (Q5_K base + Q8_0 gates) | **PPL 6.5456** | Beats Q5_K_M (6.57) | | SHQv3-C (IQ4_XS FFN + Q8_0 attn) | PPL 6.6534 | +0.11 PPL vs SHQv2 | | SHQv3-D (Q5_K + IQ4_XS on low-imp) | Not tested | Predicted between SHQv2 and v3-C | **150K importance threshold**: IQ4_XS safe only on tensors with importance **Note:** The chat template is from Unsloth (`chat_template.jinja`). For agent/coding tasks, `temp 1` + `top_k 20` + `top_p 0.95` works well. Use `-c 65536` for long context tasks. > **VRAM usage:** ~7399 MiB / 8192 MiB (90%) at `-c 65536` with flash-attn + q8_0 KV cache. Room for more context thanks to the Gated DeltaNet architecture's efficient memory footprint. > **From wepiqx:** I would love to see this fine-tune released with MTP (Multi-Token Prediction) support — would make for an even better quantization. If Empero adds MTP, I'll update this quant with a specialized MTP draft model for faster inference. ### Ollama 1. Create a `Modelfile`: ``` FROM ./SHQ8-Q5_K_M.gguf TEMPLATE """{{ .System }} {{ .Prompt }}""" SYSTEM """You are a helpful coding assistant.""" PARAMETER num_ctx 8192 PARAMETER temperature 1 PARAMETER top_k 20 PARAMETER top_p 0.95 ``` 2. Build and run: ```bash ollama create qwable-SHQ8 -f Modelfile ollama run qwable-SHQ8 ``` > **Important — sampling (wepiqx):** I strongly recommend **not using `repeat_penalty`** (keep at 1.0 / off). Testing shows it degrades output quality on this quant. Use `temperature 1` (general) or `temperature 0.6` (precise coding tasks). `top_k 20` recommended. The model's native repetition handling is deliberately tuned and doesn't need penalizing. ### LM Studio 1. Open LM Studio 2. Drag `SHQ8-Q5_K_M.gguf` into the app 3. Set GPU Offload to 99 layers 4. Enable flash-attention 5. Set context length to 8192+ 6. **Crucial — disable `repeat_penalty`** (set to 1.0 or off). Use `temperature 1` (general) or `0.6` (precise tasks), `top_k 20` 7. Start chatting ## Coding Examples These models generate full, working HTML/CSS/JS websites in a single pass. Both examples used the same prompt at `temperature 1`: > "I'm a dev, my audience is youth. I like a creative/tech style. Write the full website code. This HTML will be our foundation." ### SHQ-OptA — [SHQ8_temp-1.html](SHQ8_temp-1.html) A complete, styled developer portfolio in **324 lines**: - CSS variables, background grid effect - Typewriter animation (pure JS) - Card grid with hover glow - Terminal-style contact form - Responsive design (mobile tweaks) - JetBrains Mono font integration ### SHQ8-v2 — [SHQ8-v2_temp-1.html](SHQ8-v2_temp-1.html) Same prompt, ***355 lines*** — cleaner code, no external dependencies: - Ambient floating orb background (pure CSS) - Typewriter effect + 3D tilt card (mousemove parallax) - Code window with syntax-highlighted preview - Custom cursor glow trail - Responsive design with mobile-safe tilt disable - **Zero external dependencies** — no Google Fonts, everything shipped inline ## Lessons Learned 1. **Imatrix from same dataset as training = gold.** Importance signals are unusually clean. 2. **IQ4_XS on FFN gate/up destroys non-QAT models.** 150K importance threshold is real. 3. **CODER experiments transferred 1:1.** Same architecture, different fine-tune, same optimal strategy. 4. **Q5_K_M + imatrix > Q6_K.** The per-row refinement compensates for lower base bpw. 5. **Manual --tensor-type overrides beat imatrix promotions.** Q8_0 stays Q8_0. 6. **Temperature 2 + SHQ-OptA gives incredible coding output.** User reported impressive results. 7. **Real imatrix data matters more than README claims.** The actual importance matrix had 2× lower values and different layer ordering than documented, leading to a different optimal strategy. 8. **ssm_out is quantization noise** at 15-155 importance (10,000× below top tensors). Safe for IQ4_XS or even IQ2_XXS. 9. **L0 and deep layers (26-31) need Q8_0; middle layers (1-25) suffice at Q6_K.** Attention position matters more than attention type. ## Future Work - Try IQ2_XXS on ssm_out (importance 15-155 — near zero) - Profile speed impact of IQ4_XS tensors on GTX 1070 (Pascal lacks IQ4 hardware) ## Files | File | Size | Description | |---|---|---| | `Qwable-9B-Claude-Fable-5-bf16.gguf` | 17 GB | BF16 source | | `Qwable-9B-Claude-Fable-5.imatrix.gguf` | 5 MB | Importance matrix (mradermacher) | | `Qwable-9B-Claude-Fable-5-i1-Q6_K.gguf` | 6.9 GB | Q6_K baseline | | `SHQ8-Q5_K_M.gguf` | 6.2 GB | **Winner — recommended for inference** | | `SHQ8-v2-Q5_K_M.gguf` | 5.7 GB | **SHQ8-v2 — hybrid Q5_K_M base with tiered precision (compact champion)** | | `SHQ8_temp-1.html` | -- | Coding example (OptA): full website at temp 1 | | `SHQ8-v2_temp-1.html` | -- | Coding example (v2): same prompt, no external deps | ## References - [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) - [Qwable-9B-Claude-Fable-5](https://huggingface.co/empero-ai/Qwable-9B-Claude-Fable-5) -...
