--- license: mit license_name: bitnet-158 license_link: https://github.com/microsoft/bitnet/blob/main/LICENSE language: - en library_name: llama.cpp tags: - bitnet - ternary - 1.58-bit - quantized - q4_k_m - gguf - llama.cpp - edge - efficient-inference - cpu - tool-calling extra_gated_fields: Name: text Email: text Country: text Organization: text I agree to the terms of BitNet: checkbox datasets: - Qapdex/agentic-foresight-actions-2k base_model: - Qapdex/SLM750-Edge-1.58-bit pipeline_tag: text-generation --- # SLM750-Edge 1.58-bit — Q4_K_M Quantized GGUF A compact, efficiently quantized BitNet b1.58 ternary model optimized for edge deployment. This repository provides a plug-and-play GGUF file ready for use with llama.cpp and its ecosystem (llama-cli, llama-cpp-python, text-generation-webui, and more). > **⚠️ IMPORTANT — PLEASE READ:** This model does **NOT** run with the standard `llama.cpp`. It requires a **patched `bitnet.cpp`** (see below). Without the patch, you will encounter `missing tensor 'blk.0.attn_sub_norm.weight'` or similar errors. The BitNet folder contains benchmark and automatic installation scripts & fixes for known problems during the installation of Python Torch and sentencepiece from python3.13/sitepackages. --- # SLM750-Edge: Distillation + GGUF [Build Colab](https://colab.research.google.com/drive/1bhAMdQF8B7glFXV3lc6aRiX16V6sdPjk?usp=sharing) # slm750_edge_gpu_package.tgz.gz [Google Drive](https://drive.google.com/drive/folders/1saghK411OS7cP29TCCftuPhKm_-8Nl7W) --- # Model Highlights | Attribute | Value | |-----------|-------| | **Architecture** | BitNet b1.58 (ternary {-1, 0, +1} weights) | | **Parameters** | ~1.4B | | **Embedding Dim** | 1,536 | | **Attention Heads** | 12 (4 KV heads, GQA) | | **Feed-Forward** | 4,096 (ReLU2 activation) | | **Context Length** | 8,192 tokens | | **Vocabulary** | 256,000 tokens (Gemma-2 tokenizer) | | **Quantization** | Q4_K_M (5.27 BPW) | | **File Size** | 873 MB | | **License** | BitNet 1.58 | ## File Description | File | Description | |------|-------------| | **`quantized_q4km.gguf`** | — Q4_K_M quantized GGUF | | ✅ **`Recommended`** | — Q4_K_M quantized GGUF [quant_fixed_from_source_gguf](https://huggingface.co/Qapdex/SLM750-Edge-1.58-bit/blob/main/README_quant_fixed_v2.gguf.md) (921 MB) | ## Quick Start ### Prerequisites [ik_llama](https://github.com/ikawrakow/ik_llama.cpp) This repository is a fork of llama.cpp with better CPU and hybrid GPU/CPU performance, new SOTA quantization types, // first-class Bitnet // support, better DeepSeek performance via MLA, FlashMLA, fused MoE operations and tensor overrides for hybrid GPU/CPU inference, row-interleaved quant packing, etc. ### Building llama-cpp with BitNet Support ```bash https://copilot.microsoft.com/shares/pages/kaesS7TdPccnaLsu4iQ9T ``` - **llama.cpp** built from source with BitNet support (commit `52b3df002` or later), OR - **llama-cpp-python** v0.3.x+ ### Usage with llama-cli ```bash # Basic text generation ./llama-cli -m quantized_q4km.gguf \ -p "Explain quantum computing in simple terms" \ -n 256 \ -t 4 \ --temp 0.7 \ --top-p 0.9 # Chat mode ./llama-cli -m quantized_q4km.gguf \ -p "You are a helpful assistant." \ --chat-template gemma \ -n 512 \ -t 4 ``` ### Usage with llama-cpp-python (Python) ```bash from llama_cpp import Llama llm = Llama( model_path="quantized_q4km.gguf", n_ctx=8192, n_threads=4, verbose=False, ) output = llm( "What is the meaning of life?", max_tokens=256, temperature=0.7, top_p=0.9, echo=False, ) print(output["choices"][0]["text"]) ``` ### Usage with text-generation-webui 1. Place quantized_q4km.gguf in the models/ directory. 2. Launch text-generation-webui with --model quantized_q4km.gguf. 3. Select the model in the UI under the "Model" tab. ### Usage with LangChain ```bash from langchain_community.llms import LlamaCpp llm = LlamaCpp( model_path="quantized_q4km.gguf", n_ctx=8192, n_threads=4, temperature=0.7, top_p=0.9, verbose=False, ) response = llm.invoke("Write a short poem about AI.") print(response) ``` ### Performance ```bash # SLM750-Edge: A 1.58-Bit Hybrid Edge Transformer ## Architectural Mathematics & Training Protocol ### "Null-Denkzeit für Tool-Calls. >120 tok/s auf mobiler Hardware." --- ## 1. Design Philosophy SLM750-Edge combines three revolutionary advances: 1. **Hybrid Gemma-2/SmolLM2 architecture** — interleaved local+global attention with Gemma-2's logit softcapping, optimized to 750M 2. **BitNet b1.58 native quantization** — all weights constrained to {−1, 0, +1} during training, eliminating FP multiplications at inference 3. **GBNF grammar ejection** — tool-calling structure is *baked into the token distribution* so no CoT or post-processing is needed **Result:** A 750M model that fits in 142 MB at 1.58-bit (≈ 750M × 1.58 bits ÷ 8 = 148 MB), runs entirely in CPU cache on modern phone SoCs, and produces guaranteed-JSON tool calls at wire speed. --- ## 2. Architecture Blueprint ### 2.1 Macro Architecture ``` SLM750-Edge( token_embed: Vocab(256k) → dim=1536 # embedding table layers: 24 × HybridDecoderLayer # 12 global + 12 local (interleaved) norm_final: SubLN # pre-RMSNorm + activation quantization lm_head: dim=1536 → Vocab(256k) # tied weights (optional) ) ``` ### 2.2 Layer Configuration | Parameter | Value | Rationale | |-----------|-------|-----------| | `hidden_dim` | 1536 | Power-of-2 multiple for NEON/ARM dot-product efficiency | | `intermediate_dim` | 4096 | ReLU2 activation: 1536 × 8/3 ≈ 4096 (SmolLM2 ratio) | | `num_attention_heads` | 12 | 1536 ÷ 12 = 128 head_dim — optimal for FlashAttention-2 | | `num_kv_heads` | 4 | GQA ratio 3:1 — 75% KV-cache reduction vs MHA | | `head_dim` | 128 | FlashAttention-2 tile alignment (128 × 128 warp) | | `num_layers` | 24 | 12 global + 12 sliding window (interleaved) | | `sliding_window_size` | 1024 | Gemma-2 default — covers ~1K tokens of local context | | `vocab_size` | 256000 | SmolLM2 tokenizer + tool-token extensions | | `max_seq_len` | 8192 | 8K context for complex multi-step tool orchestration | **Total parameters:** ~750M (exact: 748,634,112) ### 2.3 Parameter Budget (Exact) | Component | Shape | Parameters | |-----------|-------|------------| | Embedding | 256000 × 1536 | 393,216,000 | | Per-layer Attention Q | 1536 × 1536 | 2,359,296 | | Per-layer Attention K | 1536 × 512 (GQA 4 KV heads) | 786,432 | | Per-layer Attention V | 1536 × 512 | 786,432 | | Per-layer Attention O | 1536 × 1536 | 2,359,296 | | Per-layer FFN Gate | 1536 × 4096 | 6,291,456 | | Per-layer FFN Up | 1536 × 4096 | 6,291,456 | | Per-layer FFN Down | 4096 × 1536 | 6,291,456 | | Per-layer SubLN (2x) | 2 × 1536 | 3,072 | | **Per-layer total** | — | **25,164,288** | | **24 layers total** | — | **603,942,912** | | Norm final | 1536 | 1,536 | | LM Head (tied w/ embed) | — | 0 | | **Grand total** | — | **748,634,112** ≈ 749M | --- ## 3. The BitNet b1.58 Mathematical Core ### 3.1 Ternary Weight Quantization Every linear weight tensor $W \in \mathbb{R}^{d \times k}$ is quantized during **both forward and backward passes**: $$W_q = \text{round\_clip}\left(\frac{W}{\alpha}\right) \quad \text{where} \quad \alpha = \frac{1}{n}\sum_{ij} \max(|W_{ij}|)$$ $$\text{round\_clip}(x) = \begin{cases} +1, & x > 0.5 \\ 0, & |x| \leq 0.5 \\ -1, & x 120 tok/s on Snapdragon 8 Gen 3 (4x Cortex-A720) - Decode: >60 tok/s (single token at a time, bandwidth-bound) ``` --- ## 10. Code Map | File | Purpose | |------|---------| | `bitnet_linear.py` | BitLinear layer with STE ternary quantization | | `slm750_model.py` | Full 750M hybrid architecture with SubLN | | `distill_train.py` | Knowledge distillation loop with Gemma-2 teacher | | `namespace_fix.py` | Dynamic namespace correction table builder | | `tool_grammar.gbnf` | GBNF grammar for tool-call JSON | | `gather_training_data.py` | Generate distilled dataset from LFM2 teacher | ### Key Architectural Features ```bash Component Specification Weight Precision Ternary {-1, 0, +1} (training), Q4_K_M (storage) FFN Activation ReLU2 (relu(x)2) Attention Grouped-Query Attention (GQA), 12 heads, 4 KV heads Positional Encoding RoPE (Rotary Position Embeddings) Normalization RMSNorm (epsilon = 1e-6) Logit Softcapping Attention: 50.0, Final: 30.0 (tanh-based) Context Length 8,192 tokens Quantization Format The model is quantized using Q4_K_M (4-bit K-quant, medium size): File type: LLAMA_FTYPE_MOSTLY_Q4_K_M (15) BPW: 5.27 bits per weight (including overhead) Compression ratio: ~6:1 vs. full precision Method: llama.cpp llama-quantize with --allow-requantize ``` ### Compatibility ```bash Supported Runtimes Runtime Status Notes llama.cpp (mainline) ✅ Full Requires LLM_ARCH_BITNET support (commit 52b3df002+) llama-cpp-python ✅ Full v0.3.x+ with BitNet support text-generation-webui ✅ Full Via llama.cpp backend LangChain ✅ Full Via LlamaCpp wrapper Ollama ⚠️ Manual Requires custom Modelfile; not officially supported llama-cpp.server ✅ Full OpenAI-compatible API server ``` ### Known Limitations GPU offloading is not supported for BitNet architectures in the current llama.cpp release — all inference runs on CPU. Flash Attention is not compatible with the BitNet attention implementation. Batch inference (parallel decoding) is limited by the CPU-only constraint. ## Thanks to - [llama.cpp](https://github.com/ggml-org/llama.cpp) - [BitNet b1.58](https://huggingface.co/papers/2402.17764) - Gemma-2 - SmolLM2 - [Heyneo](https://heyneo.com) - Google Ai Assistant - Microsoft Copilot
