What Is Model Quantization?

A neural network’s weights are numbers, and by default they are stored at fairly high precision — typically 16-bit floating point. Quantization represents those same weights with fewer bits, such as 8-bit or 4-bit integers (INT8 and INT4). Since a model has billions of weights, using fewer bits per weight dramatically reduces the total memory the model occupies and the bandwidth needed to move it around.

The appeal is straightforward: smaller and faster, cheaper to run. A model that would not fit on a given GPU at full precision may fit comfortably once quantized, and it often runs faster because there is less data to shuffle. Quantization is one of the highest-leverage tricks for making large models practical to deploy.

Conceptually, quantization maps a wide range of high-precision values onto a smaller set of low-precision buckets, storing an approximation of each weight rather than its exact value. The art is doing this so that the approximation preserves the model’s behavior — keeping the important distinctions between weights while discarding precision the model does not really need.

There are two broad approaches. Post-training quantization takes an already-trained model and converts its weights, which is fast and simple. Quantization-aware training factors the eventual low precision into the training process, generally preserving more quality at the cost of extra work. Modern 4-bit post-training methods such as GPTQ and AWQ have become remarkably good, retaining most of a model’s capability while quartering its memory footprint.

Integer formats such as INT8 and INT4 came first. Recent GPUs also execute low-precision floating-point formats in hardware, and because these keep a sign, an exponent and a mantissa they cope better with the occasional very large value. FP8 has two layouts: E4M3, with 4 exponent and 3 mantissa bits and a range of ±448, and E5M2, with a range of ±57,344. The vLLM project reports that FP8 halves model memory and can raise throughput by up to 1.6x with minimal accuracy impact; full FP8 compute needs an NVIDIA GPU of compute capability 8.9 or newer (Ada Lovelace, Hopper or Blackwell), while older Turing and Ampere cards can still store weights in FP8 (vLLM FP8 guide).

FP4 halves memory again. Each value uses the E2M1 layout, one sign, two exponent and one mantissa bit, which can express only a handful of magnitudes, so FP4 formats attach a shared scale to each small block of values. MXFP4, from the Open Compute Project microscaling family, shares one power-of-two scale across 32 values. NVIDIA’s NVFP4 uses 16-value blocks with a finer FP8 (E4M3) scale plus a second FP32 scale per tensor; NVIDIA puts the cost at about 4.5 bits per value and says only Blackwell GPUs support it natively (NVIDIA NVFP4 overview).

FP4 is no longer only something applied after release. OpenAI published gpt-oss with its mixture-of-experts weights already in MXFP4, which is how the 117-billion-parameter gpt-oss-120b fits on one 80 GB GPU and gpt-oss-20b runs within 16 GB of memory (gpt-oss model card).

GGUF is a file format rather than a quantization algorithm. The ggml project defines it as a format for storing models for inference with GGML and executors based on it, such as llama.cpp; one binary file carries the tensors plus metadata for the architecture, hyperparameters and tokenizer (GGUF specification). Inside that container llama.cpp offers a ladder of quantization types from Q8_0 down to 1-bit IQ variants. Its own table for Llama-3.1-8B puts Q8_0 at about 8.5 bits per weight and the widely used Q4_K_M at about 4.9, and an importance matrix computed from sample text can reduce the quality loss (llama.cpp quantize docs).

GPTQ (Frantar et al., 2022) is a one-shot post-training method that quantizes weights layer by layer using approximate second-order information, and the original paper reports 3–4 bits per weight with little accuracy loss (GPTQ paper). It needs a small calibration set and has broad kernel support in GPU serving engines.

AWQ, activation-aware weight quantization (Lin et al., 2023), rests on the finding that a small share of weights matters far more than the rest, and it picks those salient channels from activation statistics rather than from the weights themselves. Scaling them before quantizing keeps their precision (AWQ paper). Like GPTQ, it is mostly used for 4-bit weight-only models served on GPUs.

In practice few teams quantize models themselves. Publishers and the community post GGUF, AWQ, GPTQ, FP8 and NVFP4 variants of popular open-weight models, and the choice follows from the runtime and GPU you have. vLLM, for example, supports GPTQ, AWQ, bitsandbytes, FP8 and GGUF among other schemes, with a hardware compatibility matrix that decides which kernels are available on which GPU generation (vLLM quantization docs).

Quantization is not free — it is a tradeoff. Fewer bits mean a coarser approximation of the weights, and past a point that degrades the model’s accuracy. The relationship is nonlinear: moving from 16-bit to 8-bit often costs almost nothing, and 4-bit is frequently very usable, but extremely aggressive quantization can meaningfully hurt quality, especially on hard reasoning tasks.

The right precision therefore depends on the workload. A high-volume classification task may tolerate aggressive quantization for maximum efficiency, while a demanding reasoning task may warrant higher precision. Choosing the level is part of matching a model to its job and its hardware — the same optimization mindset behind sizing and routing decisions.

Quantization is often what makes on-premise AI economically viable. It lets an organization run a capable model on hardware it already owns — fitting a model onto a single GPU that would otherwise require several, or enabling deployment on edge devices. That directly lowers the cost and the infrastructure barrier to keeping AI inside your own perimeter.

It also stacks with other efficiency techniques. A model distilled into a compact student and then quantized for deployment combines savings from both. Together, these methods are what turn frontier-scale capability into models that regulated, cost-conscious enterprises can actually run themselves.

Formats you will meet on model hubs in 2026, what each one stores, and the hardware and runtime it suits. Bit counts include block-scale overhead where the source states it.