Quantization for the impatient – why int4 doesn’t speed up image generation on GPU

Long time no post… on image generation. Recently I sat down with an old codebase and put together a quantization pipeline for diffusion models out of curiosity – SDXL, FLUX, SD3, some video. Just to see quantization in practice, since almost every “tutorial” online repeats the same thing: fewer bits = faster. Makes sense, right? Less to multiply, less work for the GPU, image pops out quicker.

Except… no. And this isn’t a minor nuance, it’s a fundamental misunderstanding.

I ran measurements on a laptop with RTX 4090 mobile (16 GB VRAM), Windows 11, CUDA 12.9, Python 3.14, diffusers 0.38, torch 2.5+. Real numbers, on a real card. Without that it’s just “it runs faster on my machine”.

Why quantize at all

Let’s start from basics, because this is the first place things get muddled. Model quantization has two goals – and almost never both at once:

1. Fit the model in VRAM – weights go from fp16 (2 bytes/parameter) to int8 (1 byte) or int4 (half a byte) or fp4 (on the new RTX 5000). A 12B parameter model drops from 24 GB to 6 GB. This is the main reason anyone does it at all.
2. Sometimes speed up inference – and here’s where the minefield starts. Because “sometimes” means “only if your GPU has native tensor cores for that format”. And most formats used by quantization libraries… don’t have them.

Everything else in this article follows from point two. If you take only one thing from this post, make it this: quantize if you have to because of VRAM. Don’t quantize for speed, because it’ll almost always be slower.

Why fewer bits ≠ faster

The mechanics are simple once you see them. A GPU has physical, silicon-level MatMul instructions only for specific number formats. Everything else is software.

Everything else – nf4, int2, int1, int3, fp4 on consumer Ada – has no native MatMul on any GPU available to mortals. Which means when a quantization library “promises” you NF4, what actually happens is this:

1. Weights sit in VRAM packed (e.g. 8 NF4 weights squeezed into one int32). VRAM saved, works here.
2. Right before the multiplication… we unpack them back to fp16. Custom CUDA kernel bitwise unpack.
3. MatMul runs normally in fp16, on the same tensor cores as the fp16 baseline.

So we save VRAM (weights are packed), but we add 10-30% overhead for dequantize on every step. The lower the bits, the more bitwise operations on unpack. More operations = slower.

For fp16 → int4 the savings are 4×, dequantize is relatively cheap – sometimes it roughly breaks even. For fp16 → int2 or int1 the savings are 8×-16×, but dequantize starts costing more than the multiplication itself. Result: on 1-bit weights, inference can be 3× slower than fp16.

Benchmark table – so nobody has to take my word for it

SDXL 1024×1024, 25 steps, RTX 4090 mobile (Ada, 16 GB VRAM), same base SDXL 1.0 weights:

Look at the first two rows. bnb-int4 is exactly as fast as fp16. Because Ampere/Ada have int4 tensor cores – weights go through natively, no on-the-fly dequantize.

Now look at bnb-int8 – 3.6× slower. Why? Because bnb-int8 isn’t pure INT8 weight-only – it’s mixed precision with outlier decomposition (Tim Dettmers’ LLM.int8()). Some weights go in INT8, some in fp16, there are two separate multiplications, plus a bf16→fp16 cast between operations. It was designed for LLM stability, not speed. Works great for what it was built for, but for diffusion it’s not a good “default” choice.

HQQ-3bit – 52 seconds instead of 9. Six times slower than baseline. So to “save VRAM” you trade away 5× more time on every image. If you have 1000 images to generate, that difference gets very real very fast.

The conclusion from the table is brutal: don’t look at the bit count, look at whether your GPU has native MatMul for that format. That’s the one and only heuristic that actually works.

Five backends – what to use when

The diffusers/transformers ecosystem has five main quantization libraries. Each with its own character. Quick overview, because each deserves its own post, but you need a map:

1. bitsandbytes (bnb)

Widest support, simplest path. Supports int8 and 4-bit (NF4, INT4, FP4-E2M1). OOTB in diffusers via BitsAndBytesConfig.

• bnb-int8 – mixed precision with outlier decomposition. Stable, but slow (see table).
• bnb-nf4 – format designed for LLMs (normal weight distribution). Stable, but no tensor cores → dequant overhead.
• bnb-int4 – classic INT4 weight-only. On Ampere/Ada hits int4 tensor cores, free speed.

2. torchao

The official PyTorch path. Requires torchao≥0.7 + diffusers≥0.38. The package that will catch up to the rest of the ecosystem in 2026/2027 – but still a bit painful to use because the API keeps changing.

• torchao-int8 – Int8WeightOnlyConfig. Stable everywhere.
• torchao-int4 – Int4WeightOnlyConfig. Stable on the denoiser, unstable on the CLIP-G text encoder (more on that below).
• torchao-fp8 – Float8WeightOnlyConfig. Unstable on text encoder. OK on UNet.
• torchao-fp4 – Float8DynamicActivationInt4WeightConfig. Requires Blackwell. Won’t run on consumer Ada.

Important gotcha: in diffusers 0.38, TorchAoConfig(quant_type=...) requires an instance of AOBaseConfig, not a string. Older diffusers accepted "int8wo". If something suddenly breaks after pip upgrade – this is probably why.

3. quanto (optimum-quanto)

HuggingFace’s backend. Narrow precision support (int2, int4, int8, float8), but int2 is a unique feature – you won’t find it anywhere else. Just don’t expect it to speed anything up :)

4. HQQ (Half-Quadratic Quantization)

Calibration-free, arbitrary bit count. 1, 2, 3, 4, 8 bit. Replaces nn.Linear with HQQLinear at runtime.

Upsides: no calibration dataset (unlike GPTQ/AWQ), free choice of bits, runs on any GPU. Downsides: Python forward() path with no fused kernel (slow for low bits – see HQQ-3bit in the table), 1/2/3 bit produce visible artifacts on SDXL, 30+ second load times.

5. GGUF (community)

Format from llama.cpp, adapted by the community for FLUX/SD3 (e.g. city96/FLUX.1-dev-gguf repo).

Stability table: not everything quantizes equally well

Here’s where it gets actually interesting – because some components handle quantization fine, while others… fall apart completely. Empirical observations on SDXL:

Two observations worth noting.

First – CLIP-G hates fp8 and int4 weight-only. Why? CLIP-G has 1.4B parameters with significant dynamics outside the [-448, 448] range (the fp8 E4M3 range). Quantize the weights → lose the outliers → embeddings come out garbage → UNet computes noise → VAE outputs NaN → black image.

A black image is the classic symptom of NaN in latents. If you see it, first thought should be: “what did I break in the text encoder”. That’s where it is 90% of the time.

Second – VAE stays in fp16, always. It’s a small component (~80 MB), but the operations (convolutions, upsampling) are very sensitive to noise. There’s no point touching it – the VRAM savings are negligible and the quality degradation can be visible.

Profiler output

sdf-xl-1.0:torchao-fp4   TextToImageGenerator:create_model  not finished  N/A           N/A            N/A           N/A                               1
sdf-xl-1.0:quanto-int8   TextToImageGenerator:create_model  9.67669 sec   9.67669 sec   9.67669 sec    9.67669 sec   4458246144                        1
sdf-xl-1.0:quanto-int8   TextToImageGenerator:generate      10.05025 sec  10.05025 sec  10.05025 sec   10.05025 sec  -12623872                         1
sdf-xl-1.0:quanto-int8   TextToImageGenerator:save_results  0.02003 sec   0.02003 sec   0.02003 sec    0.02003 sec   0                                 1
sdf-xl-1.0:torchao-int8  TextToImageGenerator:create_model  11.80200 sec  11.80200 sec  11.80200 sec   11.80200 sec  3877761024                        1
sdf-xl-1.0:torchao-int8  TextToImageGenerator:generate      13.60181 sec  13.60181 sec  13.60181 sec   13.60181 sec  10059776                          1
sdf-xl-1.0:torchao-int8  TextToImageGenerator:save_results  0.01174 sec   0.01174 sec   0.01174 sec    0.01174 sec   0                                 1
sdf-xl-1.0:hqq-8bit      TextToImageGenerator:create_model  29.19757 sec  29.19757 sec  29.19757 sec   29.19757 sec  4755091456                        1
sdf-xl-1.0:hqq-8bit      TextToImageGenerator:generate      16.60006 sec  16.60006 sec  16.60006 sec   16.60006 sec  4853760                           1
sdf-xl-1.0:hqq-8bit      TextToImageGenerator:save_results  0.01172 sec   0.01172 sec   0.01172 sec    0.01172 sec   0                                 1
sdf-xl-1.0:torchao-int4  TextToImageGenerator:create_model  not finished  N/A           N/A            N/A           N/A                               2
sdf-xl-1.0:hqq-4bit      TextToImageGenerator:create_model  35.28185 sec  35.28185 sec  35.28185 sec   35.28185 sec  1573851136                        1
sdf-xl-1.0:hqq-4bit      TextToImageGenerator:generate      28.32846 sec  28.32846 sec  28.32846 sec   28.32846 sec  22056960                          1
sdf-xl-1.0:hqq-4bit      TextToImageGenerator:save_results  0.01318 sec   0.01318 sec   0.01318 sec    0.01318 sec   0                                 1
sdf-xl-1.0:bnb-int4      TextToImageGenerator:create_model  17.52835 sec  17.52835 sec  17.52835 sec   17.52835 sec  1704247296                        1
sdf-xl-1.0:bnb-int4      TextToImageGenerator:generate      13.44075 sec  13.44075 sec  13.44075 sec   13.44075 sec  -14581760                         1
sdf-xl-1.0:bnb-int4      TextToImageGenerator:save_results  0.01033 sec   0.01033 sec   0.01033 sec    0.01033 sec   0                                 1
sdf-xl-1.0:quanto-int2   TextToImageGenerator:create_model  14.87508 sec  14.87508 sec  14.87508 sec   14.87508 sec  870027264                         1
sdf-xl-1.0:quanto-int2   TextToImageGenerator:generate      24.74862 sec  24.74862 sec  24.74862 sec   24.74862 sec  13602816                          1
sdf-xl-1.0:quanto-int2   TextToImageGenerator:save_results  0.01268 sec   0.01268 sec   0.01268 sec    0.01268 sec   0                                 1
sdf-xl-1.0:hqq-2bit      TextToImageGenerator:create_model  29.01123 sec  29.01123 sec  29.01123 sec   29.01123 sec  1251397632                        1
sdf-xl-1.0:hqq-2bit      TextToImageGenerator:generate      15.55375 sec  15.55375 sec  15.55375 sec   15.55375 sec  21401600                          1
sdf-xl-1.0:hqq-2bit      TextToImageGenerator:save_results  0.01248 sec   0.01248 sec   0.01248 sec    0.01248 sec   0                                 1
sdf-xl-1.0:hqq-1bit      TextToImageGenerator:create_model  31.83030 sec  31.83030 sec  31.83030 sec   31.83030 sec  558329856                         1
sdf-xl-1.0:hqq-1bit      TextToImageGenerator:generate      34.51453 sec  34.51453 sec  34.51453 sec   34.51453 sec  -3112960                          1
sdf-xl-1.0:hqq-1bit      TextToImageGenerator:save_results  0.01281 sec   0.01281 sec   0.01281 sec    0.01281 sec   0                                 1

Overall time for each index:

Index                    Total Time     Min          Max            Average         Call Count
-----------------------  -------------  -----------  -------------  ------------  ------------
sdf-xl-1.0:quanto-int8   19.74698 sec   0.02003 sec  10.05025 sec   6.58233 sec              3
sdf-xl-1.0:torchao-int8  25.41556 sec   0.01174 sec  13.60181 sec   8.47185 sec              3
sdf-xl-1.0:hqq-8bit      45.80934 sec   0.01172 sec  29.19757 sec   15.26978 sec             3
sdf-xl-1.0:hqq-4bit      63.62349 sec   0.01318 sec  35.28185 sec   21.20783 sec             3
sdf-xl-1.0:bnb-int4      30.97943 sec   0.01033 sec  17.52835 sec   10.32648 sec             3
sdf-xl-1.0:quanto-int2   39.63639 sec   0.01268 sec  24.74862 sec   13.21213 sec             3
sdf-xl-1.0:hqq-2bit      44.57746 sec   0.01248 sec  29.01123 sec   14.85915 sec             3
sdf-xl-1.0:hqq-1bit      66.35763 sec   0.01281 sec  34.51453 sec   22.11921 sec             3

Ps. Would be interesting to write a custom kernel for a 1.5B quantization :P a 1.5B LLM is actually in progress – for now my math ran out somewhere around university :P