Qwable 9B Claude Fable 5 LWQ8 GGUF

Qwable-9B-Claude-Fable-5 — SHQ Family (Selective Hybrid Quants)

Adaptive quantization family for Qwen3.5-9B hybrid Gated DeltaNet + Full Attention architecture. Each variant uses real imatrix data to selectively assign precision per-tensor.

Model: empero-ai/Qwable-9B-Claude-Fable-5 — Claude Fable 5 reasoning fine-tune of Qwen3.5-9B.

Note: The model file SHQ8-Q5_K_M.gguf is named for HF parser compatibility. This is NOT a pure Q5_K_M quantization — it's a hybrid using Q5_K_M base with Q8_0, Q5_K, Q4_K, and F16 selectively applied across different tensor types. Check the config for exact per-tensor types.

Note: SHQ8-v2-Q5_K_M.gguf is also a hybrid — Q5_K_M base with Q8_0 on critical attention, Q6_K on mid attention, IQ4_XS on low-importance tensors. See configs/SHQ8_v2.sh for the exact per-tensor map.

Speed: ~22 tokens/sec on GTX 1070 (8 GB VRAM) with flash-attn for SHQ-OptA, ~26 tokens/sec for SHQ8-v2 (fewer Q8_0 tensors → less data through Pascal's PCIe).

I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback.

SHQ Family Tree

SHQ Series (Selective Hybrid Quantization)
│
├── SHQ6 ── Gemma 4 12B IT QAT
│   (QAT model, IQ4_XS all FFN + Q8_0 attention)
│   └── gemma4/README.md
│
├── SHQ7 ── Gemma 4 12B OBLITERATED
│   (non-QAT, gradient IQ4_NL + Q4_K + IQ4_XS + Q8_0)
│   └── gemma4/OBLITERATED_SHQ7_README.md
│
├── SHQ8 ── Qwen3.5-9B fine-tunes (non-QAT, Gated DeltaNet hybrid)
│   │
│   ├── Qwable-9B-Claude-Fable-5 (this)
│   │   ├── SHQ-OptA ★ WINNER
│   │   │   Q5_K_M base + Q8_0 on attn_gate/qkv/ssm_alpha/beta + imatrix
│   │   │   └── 6303 MiB, PPL 7.3747 — beats Q6_K in size + quality
│   │   ├── SHQ8-v2 — compact champion
│   │   │   Q5_K_M + selective Q8_0/Q6_K/IQ4_XS by imatrix
│   │   │   └── 5726 MiB, PPL 7.4559 — 18% smaller than OptA
│   │   ├── SHQ8-v3 — IQ4_NL experiment
│   │   │   Q5_K_M + all attn Q8_0 + IQ4_NL on imp ffn_down
│   │   │   └── 6041 MiB, PPL 7.4456 — 4% smaller than OptA
│   │   ├── SHQ8-v3a — all-attn Q8_0 test
│   │   │   Q5_K_M + all attn Q8_0 + ffn_down blk.20-30 Q8_0
│   │   │   └── 6301 MiB, PPL 7.4310 — same size, +0.056 vs OptA
│   │   └── Exp1 ── aborted
│   │       IQ4_XS all FFN (wrong for non-QAT)
│   │
│   └── Qwythos-9B-Claude-Mythos-5-1M
│       ├── SHQ8-OptA ★ quality champion
│       │   Same formula as OptA above
│       │   └── 6303 MiB, PPL 7.4831 — best PPL on Qwythos
│       ├── SHQ8-v2 — compact champion
│       │   Same tiered IMatrix approach as v2 above
│       │   └── 5726 MiB, PPL 7.6542 — 577 MiB less than OptA
│       └── MTP variants (built-in draft head for speculative decoding)
│           ├── M-SHQ8-MTP-OptA
│           │   OptA + blk.32 (MTP) at Q8_0
│           │   └── 6.40 GB, PPL 7.4831 — quality champion
│           └── M-SHQ8-MTP-v2
│               v2 + blk.32 (MTP) at Q8_0
│               └── 5.83 GB, PPL 7.6542 — compact champion

TL;DR

Quant	Size	PPL (ctx=1024)	vs Q6_K	vs OptA
i1-Q6_K (baseline)	7008 MiB	7.4394 ± 0.05	—	—
SHQ-OptA ★ quality	6303 MiB	7.3747 ± 0.05	best PPL	—
SHQ8-v2 ★ compact	5726 MiB	7.4559 ± 0.05	+0.017, -1282 MiB	+0.081, -577 MiB
SHQ8-v3	6041 MiB	7.4456 ± 0.05	+0.006, -967 MiB	+0.071, -262 MiB
SHQ8-v3a	6301 MiB	7.4310 ± 0.05	-0.008, -707 MiB	+0.056, -2 MiB

SHQ-OptA is the quality champion (best PPL). SHQ8-v2 is the compact champion (18% smaller than OptA at +0.081 PPL).

Key Result

Q5_K_M base + Q8_0 on attn_gate/qkv/ssm_alpha/ssm_beta + blk.31.ffn_down Q8_0 + imatrix per-row refinement.

• PPL 7.3747 vs Q6_K 7.4394 (Δ = -0.065, within margin but directionally better)
• Size 6303 MiB vs Q6_K 7008 MiB (10% smaller)
• Imatrix auto-promotes high-importance tensors (ffn_down in deep layers → Q6_K)
• Manual overrides take precedence: attn_gate/qkv → Q8_0 everywhere, blk.31.ffn_down → Q8_0

SHQ8-v2 — Real IMatrix Tiered Precision

Q5_K_M base + imatrix per-row + tiered precision by real importance data.

Real importance values from the actual Qwable-9B-Claude-Fable-5.imatrix.gguf (248 tensors, 319 chunks) revealed a 33,000× range — from blk.31 ffn_down at 323K to blk.0-5 ssm_out at 15. This enables safe aggressive quantization on low-importance tensors.

Tier	Types	Size Impact	Rationale
Q8_0	blk.0 attn (L0), blk.26-31 attn, blk.31 ffn_down, blk.31 attn_output, norms	+~155 MiB vs Q5_K	Highest importance layer edges
Q6_K	blk.1-25 attn_gate/qkv/ssm_alpha/beta	-~170 MiB vs Q8_0	Mid layers don't need Q8_0
IQ4_XS	ssm_out (all), attn_output (all), ffn_down (except blk.31)	-~350 MiB vs Q5_K	Low importance (43-21K mean)
F16	ssm_dt, ssm_a, all norms	Negligible	Tiny tensors
Q5_K_M	Everything else + imatrix per-row	Base	Proven design

Key insight — real imatrix ≠ README claims: The README's importance matrix table (1M+ for attention) was from a different file. The actual model's imatrix shows ~2× lower values and a different layer ordering (L0 dominates at 494K, not mid layers). This changes which tensors need high precision.

Size: 5,726 MiB (5.36 BPW) — 577 MiB (-9.2%) less than SHQ-OptA.

PPL: 7.4559 ± 0.05 — matches Q6_K (7.4394) within margin at 18% smaller size. Slightly higher than SHQ-OptA (7.3747) but saves 577 MiB (9.2%). A favorable size/quality trade-off.

Speed: ~26 tokens/sec on GTX 1070 — 18% faster than SHQ-OptA thanks to fewer Q8_0 tensors reducing VRAM bandwidth pressure.

Architecture

Same as Qwen3.5-9B — hybrid Gated DeltaNet + Gated Attention (3:1 pattern):

Property	Value
Layers	32 (24 DeltaNet + 8 Full Attention)
Hidden dim	4096
FFN intermediate	12288
Vocabulary	248,320
Attention-only	blk.3, 7, 11, 15, 19, 23, 27, 31
DeltaNet layers	all others

Full architecture details in SHQ-QWEN35-INSIGHTS_CODER.md.

How We Made It

Step 1: Q6_K baseline (PPL 7.4394)

Standard Q6_K with mradermacher's imatrix. Fits 8 GB VRAM at 7008 MiB.

Step 2: The IQ4_XS trap (aborted)

We tried IQ4_XS on all FFN tensors across all layers — blk.\d+.ffn_(gate|up|down)=IQ4_XS. This was a bad design for a non-QAT model: ffn_gate/up have 572K importance (far above the 150K IQ4_XS safety threshold). The run was aborted before completion.

Step 3: CODER insights rescue

Existing experiments on Qwopus3.5-9B-Coder-MTP (same architecture, different fine-tune) provided critical evidence (tested at ctx=2048):

Experiment	Result	Lesson
SHQv2 (Q5_K base + Q8_0 gates)	PPL 6.5456	Beats Q5_K_M (6.57)
SHQv3-C (IQ4_XS FFN + Q8_0 attn)	PPL 6.6534	+0.11 PPL vs SHQv2
SHQv3-D (Q5_K + IQ4_XS on low-imp)	Not tested	Predicted between SHQv2 and v3-C

150K importance threshold: IQ4_XS safe only on tensors with importance < 150K:

Tensor	Importance	IQ4_XS safe?
attn_q/k/v (Full Attn)	1,020,156	❌
attn_gate/qkv (DeltaNet)	879,647	❌
ssm_alpha/beta	879,647	❌
ffn_gate/up	572,159	❌
attn_output	127,631	✅
ffn_down	84,123	✅
ssm_out	43,393	✅

Full analysis in QUANT_HISTORY_CODER.md.

Step 4: SHQ-OptA — Winner

Chose the proven SHQv2 formula adapted for Qwable:

Base:        Q5_K_M (imatrix per-row refinement)
Attention:   Q8_0 — attn_gate, attn_qkv, ssm_alpha, ssm_beta (ALL layers)
Outlier:     Q8_0 — blk.31.ffn_down (most important tensor)
Output:      Q5_K
Token embd:  Q4_K
Norms:       F16, attn_q/k_norm Q8_0

Imatrix auto-promotes:

• ffn_down in deep layers (18-30) → Q6_K (higher importance)
• attn_v in some layers → Q6_K

Manual Q8_0 overrides are never downgraded.

Imatrix Notes

The importance matrix (Qwable-9B-Claude-Fable-5.imatrix.gguf) was computed by mradermacher from imatrix-training-full-3 — the same dataset used for fine-tuning the model itself. This gives unusually accurate importance signals.

Key findings from imatrix:

• Inverted importance: Upper layers more important (blk.31 = 12× blk.1)
• blk.31.ffn_down is the single most important tensor (3.97e+09)
• Attention = 42.5% of total importance, FFN = 30.1%, SSM = 27.4%
• Layer 1 is anomalously low (7.26e+08) — embedding boundary

Quantization Commands

SHQ-OptA

llama-quantize \
  --imatrix Qwable-9B-Claude-Fable-5.imatrix.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "output_norm.*=Q8_0" \
  --tensor-type "blk\.\d+\.attn_gate=Q8_0" \
  --tensor-type "blk\.\d+\.attn_qkv=Q8_0" \
  --tensor-type "blk\.\d+\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.\d+\.ssm_beta=Q8_0" \
  --tensor-type "blk\.31\.ffn_down=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*ssm_conv1d.*=Q8_0" \
  --tensor-type ".*ssm_dt.*=F16" \
  --tensor-type ".*ssm_a=F16" \
  --tensor-type ".*attn_norm.*=F16" \
  --tensor-type ".*post_attention_norm.*=F16" \
  --tensor-type ".*ssm_norm.*=F16" \
  Qwable-9B-Claude-Fable-5-bf16.gguf \
  Qwable-9B-Claude-Fable-5-SHQ-OptA.gguf \
  Q5_K_M

SHQ8-v2

Critical: --tensor-type rules are first-match-wins. Specific overrides must come before generic.

llama-quantize \
  --imatrix Qwable-9B-Claude-Fable-5.imatrix.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "output_norm.*=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*ssm_conv1d.*=Q8_0" \
  --tensor-type "blk\.31\.ffn_down=Q8_0" \
  --tensor-type "blk\.31\.attn_output=Q8_0" \
  --tensor-type "blk\.0\.attn_gate=Q8_0" \
  --tensor-type "blk\.0\.attn_qkv=Q8_0" \
  --tensor-type "blk\.0\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.0\.ssm_beta=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.attn_gate=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.attn_qkv=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.ssm_alpha=Q8_0" \
  --tensor-type "blk\.(26|27|28|29|30|31)\.ssm_beta=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_q\.weight=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_k\.weight=Q8_0" \
  --tensor-type "blk\.(3|27|31)\.attn_v\.weight=Q8_0" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.attn_gate=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.attn_qkv=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.ssm_alpha=Q6_K" \
  --tensor-type "blk\.([1-9]|1[0-9]|2[0-5])\.ssm_beta=Q6_K" \
  --tensor-type ".*attn_norm.*=F16" \
  --tensor-type ".*post_attention_norm.*=F16" \
  --tensor-type ".*ssm_norm.*=F16" \
  --tensor-type ".*ssm_dt.*=F16" \
  --tensor-type ".*ssm_a$=F16" \
  --tensor-type ".*ssm_out=IQ4_XS" \
  --tensor-type ".*attn_output=IQ4_XS" \
  --tensor-type ".*ffn_down=IQ4_XS" \
  Qwable-9B-Claude-Fable-5-bf16.gguf \
  SHQ8-v2-Q5_K_M.gguf \
  Q5_K_M

All Experiments

Experiment	Config	Size	PPL	vs Q6_K	Verdict
i1-Q6_K (baseline)	mradermacher imatrix	7008 MiB	7.4394 ± 0.05	—	Baseline
SHQ-OptA ★	Q5_K + Q8_0 gates + imatrix	6303 MiB	7.3747 ± 0.05	best PPL	Quality champion
SHQ8-v2 ★	Real imatrix tiered Q8_0/Q6_K/IQ4_XS	5726 MiB	7.4559 ± 0.05	-0.017,...