--- license: apache-2.0 base_model: Bahushruth/Qwen3.6-35B-A3B-abliterated-v4 tags: - abliteration - uncensored - qwen3 - moe - gguf - llama-cpp - ollama model_type: qwen3_moe pipeline_tag: text-generation datasets: - Bahushruth/abliteration-harmful-enriched --- # Qwen3.6-35B-A3B-abliterated-v4-GGUF GGUF quantizations of [Bahushruth/Qwen3.6-35B-A3B-abliterated-v4](https://huggingface.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4) for llama.cpp, Ollama, LM Studio, KoboldCPP, and other GGUF-compatible runtimes. Uncensored model — refusal behavior removed via norm-preserving abliteration (0% refusal, full capability preservation). See the [base model card](https://huggingface.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4) for method details. > **Blog post:** [Abliteration: Uncensoring LLMs via Weight Surgery](https://potatospudowski.github.io/articles/abliteration) ## Quantizations All standard quants are compatible with Ollama, LM Studio, KoboldCPP, and llama.cpp out of the box — no special flags needed. | File | Size | RAM Required | Notes | |------|------|-------------|-------| | `...-BF16.gguf` | 69 GB | 80+ GB | Full precision | | `...-Q8_0.gguf` | 37 GB | 48+ GB | Near-lossless | | `...-Q6_K.gguf` | 29 GB | 40+ GB | Very high quality | | `...-Q5_K_M.gguf` | 25 GB | 32+ GB | **Recommended for 48GB systems** | | `...-Q4_K_M.gguf` | 21 GB | 24+ GB | Good quality, fits 24GB | | `...-IQ4_XS.gguf` | 19 GB | 24+ GB | High quality 4-bit (imatrix) | | `...-IQ4_NL.gguf` | 20 GB | 24+ GB | Non-linear 4-bit (imatrix) | | `...-Q3_K_M.gguf` | 17 GB | 20+ GB | Good for 16GB VRAM GPUs | | `...-IQ3_M.gguf` | 15 GB | 20+ GB | High quality 3-bit (imatrix) | | `...-IQ3_XXS.gguf` | 14 GB | 16+ GB | Smallest usable 3-bit (imatrix) | | `...-Q2_K.gguf` | 13 GB | 16+ GB | 2-bit, quality tradeoffs | | `...-IQ2_M.gguf` | 12 GB | 16+ GB | Smallest, significant quality loss | ## MTP (Multi-Token Prediction) Qwen3.6-35B-A3B includes an MTP draft head (blk.40) for speculative decoding. All standard quants above exclude MTP for maximum compatibility. A separate BF16-MTP file is available for advanced users running `llama-server` directly: | File | Size | Notes | |------|------|-------| | `...-BF16-MTP.gguf` | ~71 GB | BF16 with MTP draft head included | To use MTP speculative decoding: ```bash ./llama-server -m Qwen3.6-35B-A3B-abliterated-v4-BF16-MTP.gguf \ --jinja --spec-type draft-mtp --spec-draft-n-max 1 -ngl 99 ``` > **Runtime compatibility:** MTP requires llama-server b9180+. Ollama does not support MTP yet. KoboldCPP 1.116.1+ and current LM Studio handle MTP fine, but many users are on older installs — the standard no-MTP quants are the safe default. ## Quantization Types Explained **K-quants** use a block-wise quantization scheme where weights are grouped into blocks and each block gets its own scale factor, preserving more precision than naive round-to-nearest quantization. | Suffix | Meaning | |--------|---------| | `Q8_0` | 8-bit uniform quantization (simplest, largest) | | `Q6_K` | 6-bit k-quant. All tensor blocks use 6-bit. | | `Q5_K_M` | 5-bit k-quant, **medium** variant. Important tensors bumped to Q6_K. | | `Q4_K_M` | 4-bit k-quant, **medium** variant. Important tensors kept at Q5_K. | | `Q3_K_M` | 3-bit k-quant, **medium** variant. Important tensors at Q4_K. | | `Q2_K` | 2-bit k-quant. Aggressive compression with quality tradeoffs. | | `IQ4_XS` | 4-bit importance-matrix quant. Uses imatrix calibration for better quality at same size. | | `IQ4_NL` | 4-bit non-linear quant. Non-uniform quantization levels tuned to weight distributions. | | `IQ3_M` | 3-bit importance-matrix quant. Medium quality. | | `IQ3_XXS` | 3-bit importance-matrix quant. Extra small — minimum viable 3-bit. | | `IQ2_M` | 2-bit importance-matrix quant. Extreme compression. | The `_M` (medium) suffix means a **mixed-precision** strategy: less important layers get the headline bit-width while critical layers (attention output, embeddings) get one level higher. This gives noticeably better quality than pure `_S` (small) variants at only ~5-10% size increase. **IQ quants** use importance-matrix calibration (computed from wikitext-2) to allocate more precision to weights that matter most for model output quality. They achieve better perplexity than K-quants at the same file size. ## Quickstart — Ollama ```bash # Recommended for Apple Silicon 48GB+ (M4 Pro, M4 Max, etc.) ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:Q5_K_M # For 24GB systems ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:Q4_K_M # For 10GB VRAM (RTX 3080) ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:IQ3_XXS ``` ## Usage — llama.cpp ```bash huggingface-cli download Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF \ Qwen3.6-35B-A3B-abliterated-v4-Q5_K_M.gguf --local-dir . # Basic usage ./llama-cli -m Qwen3.6-35B-A3B-abliterated-v4-Q5_K_M.gguf \ -p "You are a helpful assistant." \ --chat-template chatml -cnv ``` ## Important Notes - This is a MoE model (256 experts, 8 active per token). Despite 35B total params, only ~3B are active — efficient for its capability level. - All standard quants work with Ollama, LM Studio, KoboldCPP, and llama.cpp without any special flags. - IQ quants (IQ4_XS, IQ3_M, etc.) use imatrix calibration for better quality-per-bit. - Tested and confirmed working on M4 Pro 48GB with Q5_K_M, and RTX 3080 10GB with IQ3_XXS. ## Conversion Details - Converter: `llama.cpp/convert_hf_to_gguf.py --no-mtp` - Quantizer: `llama-quantize` - imatrix: wikitext-2, 100 chunks, 16 threads - Infrastructure: Modal (CPU-only, 16 cores, 128GB RAM) ## Disclaimer This model has had safety guardrails removed. Released for research purposes. The creator assumes no responsibility for downstream use.
