Gemma 4 12b It NVFP4

--- license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license pipeline_tag: image-text-to-text base_model: google/gemma-4-12B-it tags: - gemma4 - unsloth - gemma - google - gemma4_unified --- # Read our How to [Run NVFP4 quants Guide!](https://unsloth.ai/docs/basics/nvfp4) <div> <p style="margin: 0 0 0px 0; margin-top: 0px;"> <em>See <a href="https://unsloth.ai/docs/basics/unsloth-dynamic-v2.0-gguf">Unsloth Dynamic 2.0 GGUFs</a> for our quantization benchmarks.</em> </p> <div style="display: flex; gap: 5px; align-items: center; margin-bottom: 0px;"> <a href="https://github.com/unslothai/unsloth/"> <img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="133"> </a> <a href="https://discord.gg/unsloth"> <img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173"> </a> <a href="https://unsloth.ai/docs/models/gemma-4"> <img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143"> </a> </div> <ul style="margin: 0;"> <li>Gemma 4 12B can now be run and fine-tuned in <a href="https://unsloth.ai/docs/new/studio">Unsloth Studio</a>. <a href="https://unsloth.ai/docs/models/gemma-4">Read our guide</a>.</li> <li>See all versions of Gemma 4 (GGUF, 16-bit etc.) <a href="https://huggingface.co/collections/unsloth/gemma-4">in our collection</a>.</li> <li>Example of Gemma 4 E4B (4-bit GGUF) running in Unsloth Studio with tool-calling:</li> </ul> </div> ## Run in vLLM ```shell uv venv unsloth-nvfp4-env --python 3.13 source unsloth-nvfp4-env/bin/activate uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" --torch-backend=auto ``` ```shell vllm serve unsloth/gemma-4-12b-it-NVFP4 ``` Do not use the Marlin backend (around 2x slower); let vLLM auto-select the NVFP4 kernel. <img width="600" alt="gemma 4 in unsloth studio" src="https://cdn-uploads.huggingface.co/production/uploads/62ecdc18b72a69615d6bd857/BTmv3pr-QQ8ZMxxX3Ofnf.gif" /> --- <div align="center"> <img src=https://ai.google.dev/gemma/images/gemma4_banner.png> </div> <p align="center"> <a href="https://huggingface.co/collections/google/gemma-4" target="_blank">Hugging Face</a> | <a href="https://github.com/google-gemma" target="_blank">GitHub</a> | <a href="https://blog.google/innovation...

Source context: 43579 downloads · 24 likes · Pipeline image-text-to-text · Repo unsloth/gemma-4-12b-it-NVFP4