Qwen3 Vl 4b Heretic

Available formats

This model is published across three repos. Pick the one that matches your runtime.

The Docker setup, scripts and configs that produced these files are in Heretic Docker.

Why this variant?

Several Heretic trials were run against Qwen3-VL-4B and all of them reach 100% HarmBench ASR, up from 30.8% on the base. This build was picked because, with safety tied, it wins on the tie-breakers:

See the full report for the comparison.

Benchmark deltas across the Heretic variants

Files

* INT4 W4A4 uses ComfyUI's convrot_w4a4 path (ComfyUI 0.30.0+). Layers with incompatible dimensions are kept in bf16.

† NVFP4 also runs on older GPUs through software dequantisation (tested on an RTX 4090). Native FP4 tensor cores need SM100+ (RTX 5090/5080).

Quality: All quantised variants use SVD-guided learned rounding (AdaRound via convert-to-quant), which optimises each weight's rounding direction to minimise output reconstruction error, noticeably higher fidelity than naive round-to-nearest quantisation.

Which quant should I pick?

• bf16 (8.3 GB) is full precision. Use it when you have the VRAM and want zero quantisation loss.
• INT8 / ConvRot (4.6 GB, any GPU) is the recommended pick. Learned rounding keeps it near-lossless, and it runs on any Ampere+ GPU without needing Blackwell.
• INT4 / W4A4 ConvRot (2.7 GB, any GPU*) is the smallest. Hadamard-rotated 4-bit signed INT. Layers with incompatible dimensions stay in bf16 automatically.
• FP8 E4M3 (4.5 GB, Ada+) is fast and small, a good fit for RTX 4090 / RTX 4500 and up.
• NVFP4 (2.9 GB) is the smallest float format. Native FP4 on Blackwell, with software dequant on older GPUs.
• MXFP8 (4.7 GB, Blackwell) handles dynamic range better than per-tensor FP8 thanks to E8M0 block scales.

Usage

1. Download one of the files above.
2. Place it in ComfyUI/models/text_encoders/.
3. Use the matching loader node in your workflow.

With Krea 2

Krea 2 is an image-generation model that runs on a Qwen3VL-4B text encoder. Because these checkpoints are the same Qwen3-VL-4B architecture (abliterated), you can drop one in as the encoder in a Krea 2 ComfyUI workflow to give it an uncensored text encoder. The fp8 checkpoint is the closest match to the stock qwen3vl_4b_fp8_scaled.safetensors; use int8 or bf16 if you want higher fidelity and have the VRAM.

Quantisation format details

FP8 (E4M3) is per-tensor scaled quantisation with learned rounding via convert-to-quant. It runs on Ada (RTX 4090) and newer, giving a good balance of size and quality.

INT8 (ConvRot row-wise) uses group-wise Hadamard rotation (ConvRot) to spread activation outliers, followed by per-row symmetric INT8 with learned rounding. For each weight tensor, an SVD-guided gradient descent loop (Prodigy optimiser) learns the rounding direction that minimises output error, which keeps INT8 quality near lossless. It runs on any modern GPU (Ampere+) with no Blackwell requirement.

INT4 (W4A4 ConvRot) is 4-bit signed INT with mandatory group-wise Hadamard rotation. The smallest variant at about a third of bf16. Layers whose dimensions are not divisible by the ConvRot group size are automatically kept in bf16. Uses ComfyUI's convrot_w4a4 path (ComfyUI 0.30.0+).

NVFP4 (E2M1) is 4-bit floating point with double quantisation (a per-tensor f32 scale plus a per-block FP8 scale, block size 16) and learned rounding. It is about three times smaller than bf16 and loads natively in ComfyUI without plugins. Blackwell GPUs (RTX 5090/5080, SM100+) use native FP4 tensor cores for best speed, while older GPUs fall back to software dequantisation (tested on an RTX 4090).

MXFP8 is microscaling FP8 (the OCP MX standard). It stores FP8 E4M3 data with E8M0 (power-of-two) per-block scales on 32-element blocks, with learned rounding. It needs SM100+ (Blackwell).

How these were made

Produced with Heretic Docker, which wraps:

• Heretic for the abliteration
• convert-to-quant for FP8, INT8 (ConvRot), INT4 (W4A4 ConvRot) and MXFP8 quantisation
• comfy_kitchen for the NVFP4/MXFP8 CUDA kernels

Limitations

• Inherits every limitation of the base Qwen3-VL-4B-Instruct model.
• Abliteration reduces refusals but does not remove them completely.
• NVFP4 and MXFP8 run best on Blackwell GPUs; NVFP4 also runs on older GPUs through software dequantisation.
• INT8 and INT4 ConvRot may leave out some vision encoder weights because of non-standard tensor dimensions; those layers stay in bf16 automatically.
• Inherits the abliteration caveats: TruthfulQA and GSM8K sit measurably lower than the base model (see the report).

Acknowledgements

• Qwen team for Qwen3-VL-4B-Instruct
• Heretic by p-e-w, the abliteration tool
• convert-to-quant by silveroxides for INT8 ConvRot, FP8 scaling and MXFP8 quantisation
• comfy_kitchen by Comfy-Org for the NVFP4/MXFP8 CUDA kernels
• Abliterlitics for the variant forensics

Disclaimer

This model has had its safety alignment removed. It complies with harmful requests, including content related to violence, illegal activities and other harmful behaviour. Use it responsibly and in line with the laws and regulations that apply to you. The authors do not condone or encourage using this model for harmful purposes.