Gemma 4 E4B Turboquant

[!TIP] KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0 (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or -ctk q4_0 -ctv q4_0 (~quarter memory, ≈7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1).

The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

gemma-4-E4B-TurboQuant

TurboQuant KV cache compression for google/gemma-4-E4B.

This is a documentation repository that explains how to combine gemma-4-E4B's weights with TurboQuant inference-time KV cache compression. No weights are stored here — use the base model directly and apply TurboQuant via the Python package or llama.cpp fork.

Hardware compatibility

What is this?

KV cache compression reduces the memory used by the attention cache during inference. Unlike weight quantization (which is baked into the GGUF/MLX file), KV cache compression is applied at runtime — so the same base weights can be used with or without compression.

Both can be combined for maximum efficiency.

Quickstart

Option A — Python / transformers

Install the turboquant package:

pip install turboquant

Then use it with the base model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from turboquant import TurboQuantCache

tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-E4B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-E4B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Apply TurboQuant to the KV cache
cache = TurboQuantCache(bits=4)  # or bits=2 for more aggressive compression

inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    past_key_values=cache,
    use_cache=True,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Option B — llama.cpp / LM Studio / Ollama (with fork)

TurboQuant KV cache types (planar3) are not in upstream llama.cpp. They require:

• llama-cpp-turboquant fork

Once built:

llama-cli -m gemma-4-E4B.gguf \
  --cache-type-k planar3 --cache-type-v planar3 \
  -ngl 99 -fa \
  -p "Hello"

For standard runtimes (LM Studio, Ollama, upstream llama.cpp), use conventional KV cache types (q8_0, q4_0). You lose the TurboQuant-specific benefits but keep GGUF weight quantization.

Model Specifications

What is TurboQuant?

TurboQuant (ICLR 2026) applies random orthogonal rotations followed by optimal scalar quantization to the KV cache. Bit-identical prefill logits at 4-bit, up to 4-8× memory savings for long sequences.

Benchmarks (from the TurboQuant repository, Llama 3.1 8B on RTX 5090 — results vary by model and hardware):

• 4-bit KV cache: bit-identical prefill logits
• ~1.4-1.7× speedup on Apple Silicon
• Up to 8× KV memory savings

Benchmarks are from the TurboQuant repository using Llama 3.1 8B. Performance on gemma-4-E4B will differ. Please open a discussion if you have independent results.

Current Ecosystem Support

Pre-quantized weight variants

If you want combined weight + KV cache compression, majentik hosts pre-quantized versions:

• MLX (Apple Silicon)
• GGUF (llama.cpp / Ollama / LM Studio)

See Also

• RotorQuant GitHub
• TurboQuant paper (arXiv 2504.19874)
• llama-cpp-turboquant fork
• Base model: google/gemma-4-E4B
• gemma-4-E4B announcement

Variants in this family

(Showing 13 sibling variants under majentik/gemma-4-e4b-*. The current variant — TurboQuant — is bolded.)