Gemma 4 E4B It Sdnq Dynamic 8bit

--- library_name: transformers license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license base_model: - google/gemma-4-E4B-it base_model_relation: quantized tags: - sdnq - gemma pipeline_tag: any-to-any --- # Gemma 4 E4B it SDNQ Dynamic INT8 This is an int8 quantized version of [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it) using [SDNQ](https://github.com/Disty0/sdnq) (SD.Next Quantization) with the dynamic option, Hadamard Rotation and quantized embeddings. Note: You need SDNQ v0.2.1 or superior (the quantized embeddings + Hadamard path requires the `nn.Embedding` fix that landed in 0.2.1). ## Usage Load it with `transformers` like the original model. You need to `import sdnq` first so it registers into the transformers quantizer registry — without it, loading fails with a size-mismatch error: ```python import sdnq # registers the SDNQ quantizer into transformers import torch from transformers import AutoModelForImageTextToText, AutoProcessor model_id = "OzzyGT/gemma_4_E4B_it_sdnq_dynamic_8bit" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda") messages = [{"role": "user", "content": [{"type": "text", "text": "Explain quantization in one sentence."}]}] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" ).to(model.device) out = model.generate(**inputs, max_new_tokens=256) print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) ```

Source context: 16 downloads · 0 likes · Pipeline any-to-any · Library transformers · Repo OzzyGT/gemma_4_E4B_it_sdnq_dynamic_8bit

Source context: 68 downloads · 0 likes · Pipeline any-to-any · Library transformers · Repo OzzyGT/gemma_4_E4B_it_sdnq_dynamic_8bit