Gemma 4 31B It NVFP4

--- license: apache-2.0 license_link: https://ai.google.dev/gemma/docs/gemma_4_license pipeline_tag: image-text-to-text base_model: google/gemma-4-31B-it tags: - gemma4 - unsloth - gemma - google --- # Read our How to [Run NVFP4 quants Guide!](https://unsloth.ai/docs/basics/nvfp4) <div> <p style="margin-top: 0;margin-bottom: 0;"> <em>See <a href="https://unsloth.ai/docs/basics/unsloth-dynamic-v2.0-gguf">Unsloth Dynamic 2.0 GGUFs</a> for our quantization benchmarks.</em> </p> <div style="display: flex; gap: 5px; align-items: center; margin-bottom: 0px;"> <a href="https://github.com/unslothai/unsloth/"> <img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="133"> </a> <a href="https://discord.gg/unsloth"> <img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173"> </a> <a href="https://unsloth.ai/docs/models/gemma-4"> <img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143"> </a> </div> </div> Gemma 4 can now be run and fine-tuned in [Unsloth Studio](https://unsloth.ai/docs/new/studio). [Read our guide](https://unsloth.ai/docs/models/gemma-4). See all versions of Gemma 4 (GGUF, 16-bit etc.) [in our collection](https://huggingface.co/collections/unsloth/gemma-4). ## Run in vLLM ```shell uv venv unsloth-nvfp4-env --python 3.13 source unsloth-nvfp4-env/bin/activate uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" --torch-backend=auto ``` ```shell vllm serve unsloth/gemma-4-31B-it-NVFP4 ``` Do not use the Marlin backend (around 2x slower); let vLLM auto-select the NVFP4 kernel. --- <div align="center"> <img src=https://ai.google.dev/gemma/images/gemma4_banner.png> </div> <p align="center"> <a href="https://huggingface.co/collections/google/gemma-4" target="_blank">Hugging Face</a> | <a href="https://github.com/google-gemma" target="_blank">GitHub</a> | <a href="https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/" target="_blank">Launch Blog</a> | <a href="https://ai.google.dev/gemma/docs/core" target="_blank">Documentation</a> <br> <b>License</b>: <a href="https://ai.google.dev/gemma/docs/gemma_4_license" target="_blank">Apache 2.0</a> | <b>Authors</b>: <a href="https://deepmind.google/models/gemma/" target="_blank">Googl...

Source context: 32730 downloads · 32 likes · Pipeline image-text-to-text · Repo unsloth/gemma-4-31B-it-NVFP4