--- license: mit tags: - stable-diffusion - stable-diffusion-diffusers inference: false --- # SDXL-VAE-FP16-Fix SDXL-VAE-FP16-Fix is the [SDXL VAE](https://huggingface.co/stabilityai/sdxl-vae)*, but modified to run in fp16 precision without generating NaNs. | VAE | Decoding in `float32` / `bfloat16` precision | Decoding in `float16` precision | | --------------------- | -------------------------------------------- | ------------------------------- | | SDXL-VAE | ✅  | ⚠️  | | SDXL-VAE-FP16-Fix | ✅  | ✅  | ## 🧨 Diffusers Usage Just load this checkpoint via `AutoencoderKL`: ```py import torch from diffusers import DiffusionPipeline, AutoencoderKL vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16) pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", vae=vae, torch_dtype=torch.float16, variant="fp16", use_safetensors=True) pipe.to("cuda") refiner = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-1.0", vae=vae, torch_dtype=torch.float16, use_safetensors=True, variant="fp16") refiner.to("cuda") n_steps = 40 high_noise_frac = 0.7 prompt = "A majestic lion jumping from a big stone at night" image = pipe(prompt=prompt, num_inference_steps=n_steps, denoising_end=high_noise_frac, output_type="latent").images image = refiner(prompt=prompt, num_inference_steps=n_steps, denoising_start=high_noise_frac, image=image).images[0] image ```  ## Automatic1111 Usage 1. Download the fixed [sdxl.vae.safetensors](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/resolve/main/sdxl.vae.safetensors?download=true) file 2. Move this `sdxl.vae.safetensors` file into the webui folder under `stable-diffusion-webui/models/VAE` 3. In your webui settings, select the fixed VAE you just added 4. If you were using the `--no-half-vae` command line arg for SDXL (in `webui-user.bat` or wherever), you can now remove it (Disclaimer - I haven't tested this, just aggregating various instructions I've seen elsewhere :P PRs to improve these instructions are welcomed!) ## Details SDXL-VAE generates NaNs in fp16 because the internal activation values are too big:  SDXL-VAE-FP16-Fix was created by finetuning the SDXL-VAE to: 1. keep the final output the same, but 2. make the internal activation values smaller, by 3. scaling down weights and biases within the network There are slight discrepancies between the output of SDXL-VAE-FP16-Fix and SDXL-VAE, but the decoded images should be [close enough for most purposes](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/discussions/7#64c5c0f8e2e5c94bd04eaa80). --- \* `sdxl-vae-fp16-fix` is specifically based on [SDXL-VAE (0.9)](https://huggingface.co/stabilityai/sdxl-vae/discussions/6#64acea3f7ac35b7de0554490), but it works with SDXL 1.0 too
