Krea2 Comfy Int4 Mixed

--- license: mit language: - en base_model: - krea/Krea-2-Turbo --- Since we have convrot int4 and int8, why not mix it up? Int4 kinda sucks, int8 is kinda slow. First we steal the recipe from: https://huggingface.co/LAXMAYDAY/Krea-2-Turbo-int4-tensorwise-mixed They didn't list it and made me hunt.. but oh well. It's here now. https://huggingface.co/Lockout/krea2-comfy-int4-mixed/blob/main/int4-mixed-recipie-int4.txt So how to quantize it? https://github.com/Starnodes2024/comfyui-starnodes-modelconverter/commit/fc114e0931dd8da458ec49c60b34a4fe44dab5e7 But it's a piece of shit and it doesn't follow the profile.. plus it casts things to half and BF16 blindly. Bad job. Here's a fix. https://huggingface.co/Lockout/krea2-comfy-int4-mixed/blob/main/star_model_converter_pro.py Did you say profile? https://huggingface.co/Lockout/krea2-comfy-int4-mixed/blob/main/int4-mixed-glm.json It's tested on krea2. But I'm sure someone will come up with a new and exciting way to make some int8/int4 mixed models with trial and error. That's really all there is to it. Kinda int4 speeds at quality somewhere between int8 and int4. Less body horrors. I will upload some quants I converted too. bonus: Script to upcast int8 to BF16 in case they don't release the full model. https://huggingface.co/Lockout/krea2-comfy-int4-mixed/blob/main/glm-dequant.py Must be run from the comfy main folder. 2080ti with sage attention and compiled: 8/8 [00:13<00:00, 1.63s/it] Prompt executed in 21.91 seconds 8/8 [00:07<00:00, 1.11it/s] Prompt executed in 7.58 seconds 8/8 [00:07<00:00, 1.11it/s] Prompt executed in 7.58 seconds vs 9.5s for pure int8 and 6.5s for pure int4. New: https://huggingface.co/Lockout/krea2-comfy-int4-mixed/blob/main/int4_balanced.json Slightly bigger model 9.6-10.3gb, less body horrors. Kung fu the search continues. Tried some approaches to take metrics. You capture int8 activations first and then you point metrics scripts at them. Either by mean error or RMSE scores. python claude_scorer_v3.py \ --model "/gonzalomoKrea2_v10.safetensors" \ --activations-dir "/outputs/star_activations/capture1" \ --out ./profiles/krea_real_activation_v3.json \ --device cuda --mean-ceiling 0.01 --max-ceiling 0.15 python glm_score_rmse.py --model "/gonzalomoKrea2_v10.safetensors" \ --activations-dir "/outputs/star_activations/capture1" \ --out ./profiles/krea_real_rmse4....

Source context: 0 downloads · 6 likes · Repo Lockout/krea2-comfy-int4-mixed