GGUF Quantization Levels Explained: Q4_K_M vs Q5_K_M vs Q8, IQ Quants and GGUF vs MLX

GGUF is the single-file model format of llama.cpp, Ollama and LM Studio; the suffix on the file name is its quantization level. Q4_K_M stores about 4.9 bits per weight, Q8_0 about 8.5. In llama.cpp's Llama 3 8B test, Q4_K_M is 4.58 GiB and raises perplexity by 0.18; Q8_0 is 7.96 GiB and raises it by 0.003. Default to Q4_K_M; prefer Q5_K_M or Q6_K for coding and tool calling.

Anyone who downloads a local model meets the same wall of file names: Q4_K_M, Q5_K_S, Q6_K, Q8_0, IQ4_XS, UD-Q4_K_XL. They all hold the same model. They differ in how many bits each weight is stored with, and so in size, memory use and accuracy.

This page explains the names, the arithmetic and the measured quality loss, and ends with a choice by memory and task. For the general idea of quantization itself, start with our short definition of model quantization; this page is only about the GGUF levels. The figures come from llama.cpp’s documentation and source code and from the model repositories named in each section, all read on 14 November 2026.

What is GGUF?

GGUF is the file format of the GGML library and of llama.cpp, the C and C++ inference project built on it. The specification describes a binary format designed for fast loading, with these aims:

• Single-file deployment. Weights, tokenizer and settings travel in one file.
• Full information. Everything needed to load the model is inside it.
• Extensible metadata. Settings are typed key-value pairs, so new fields do not break old files. This is the main change from its predecessors GGML, GGMF and GGJT.
• Memory mapping. The file can be loaded with mmap.

Models are trained in PyTorch or a similar framework, published as safetensors, and then converted. llama.cpp’s convert_hf_to_gguf.py writes a 16-bit GGUF, and llama-quantize turns that into the smaller files most people download. Ollama and LM Studio run GGUF files, and the format has spread beyond language models: image and video transformers such as LTX 2.5 now ship as community GGUF builds too.

A GGUF file is not automatically quantized. BF16 and F16 files hold the original 16-bit weights. The quantization level is a property of the file, shown in its name.

How to read a GGUF quantization name

Take Qwen3.8-27B-Q4_K_M.gguf. The part before the last hyphen is the model. The part after it is the encoding.

The k-quants date from a llama.cpp pull request by ikawrakow, merged on 5 June 2023, and its description is still the clearest account of what the letters mean. The base types are:

• Q4_K: 4-bit weights in super-blocks of 8 blocks of 32 weights, with 6-bit scales and minimums, which works out at 4.5 bits per weight.
• Q5_K: the same structure at 5 bits, 5.5 bits per weight.
• Q6_K: 6-bit weights in super-blocks of 16 blocks of 16 weights with 8-bit scales, 6.5625 bits per weight.

The file-level names are mixes of those types. Per the same description, Q4_K_S uses Q4_K for all tensors, while Q4_K_M uses Q6_K for half of the attention value and feed-forward output tensors and Q4_K for the rest. Q5_K_M follows the same pattern over Q5_K. That is why a “4-bit” Q4_K_M file averages nearly 4.9 bits.

Q8_0 is the odd one out: a plain 8-bit round-to-nearest quant in blocks of 32 weights with one scale each. It is simple and close to lossless.

Bits per weight and file size: the maths

The rule of thumb:

file size (GB) ≈ parameters (billions) × bits per weight ÷ 8

llama.cpp’s quantize documentation gives the effective bits per weight for each level, measured on Llama 3.1 8B:

The same README shows smaller files generating faster on its test machine, 71.9 tokens per second at Q4_K_M against 50.9 at Q8_0 and 29.2 at F16, though it does not name the hardware.

Worked example: Qwen3.8-27B

Unsloth’s BF16 GGUF of Qwen3.8-27B is 54.66 GB across two shards. At two bytes per weight, that is about 27.3 billion weights. Apply the rule:

The estimate lands within 1% each time, so you can size any model before downloading it. Our VRAM calculator does this sum and adds context memory.

One warning: the label is a recipe, not a guarantee of identical bytes. On the same day, ggml-org’s Q4_K_M file for this model was 19.4 GB, about 2.6 GB larger than LM Studio’s. We did not open the files to find out why. llama.cpp lets a publisher keep chosen tensors at a higher type with flags such as --tensor-type and --output-tensor-type, and the GGUF naming notes say extra prediction heads can be stored inside the main file. Compare sizes, not just names.

GGUF Q4 vs Q8: how much quality do you lose?

llama.cpp publishes a scoreboard for Llama 3 8B in its perplexity documentation, run on one RTX 4090 against the Wikitext-2 test set. Perplexity for the 16-bit model was 6.233. KL divergence measures how far the quantized model’s next-token probabilities drift from the 16-bit model’s, with 0 meaning identical. Rows are without an importance matrix.

Four things stand out.

Loss is not linear. From Q8_0 to Q4_K_M the file shrinks by 42% and perplexity rises by 2.8%. From Q4_K_M to Q2_K the file shrinks by a further 35% and perplexity rises by more than half. The knee of the curve sits between 4 and 3 bits.

K-quants beat the old types at the same size. Q4_K_S and Q4_0 are within 0.03 GiB of each other, and Q4_0 has about 1.7 times the perplexity increase. There is rarely a reason to pick a _0 or _1 file below 8 bits.

An importance matrix helps most at low bits. With one, the same table shows Q4_K_M at +0.151 instead of +0.175, Q3_K_M at +0.503 instead of +0.657 and Q2_K at about +2.42 instead of +3.52.

The “same top token” column is the practical one. At Q4_K_M the quantized model would choose a different most-likely token from the 16-bit model about one time in twelve. In prose that usually means a synonym. In code or JSON it can mean a different result.

Two cautions on reading too much into it. The documentation itself says llama.cpp’s numbers are not directly comparable with other projects’, and this is one 8B model from 2024. Newer evidence points the same way, though. Meta’s Muse Glimmer model card, from August 2026, reports that its 19.65 GB dynamic k-quant loses 0.2% and its 16.76 GB build 1.0%, averaged over 15 benchmarks, against the full-precision 30B model. That is a vendor’s measurement of its own files.

For the organisational view, what to test before a quantized build goes into service and how to record it, see our guide to weighing quantization against GPU cost on premises.

Q4_K_M vs Q5_K_M vs Q6_K vs Q8_0

Put side by side for one 8B model, using the two llama.cpp tables above:

Each step up removes about half or more of the remaining divergence. Q5_K_M is the best value step: 16% more memory for a threefold drop in divergence. Beyond Q6_K the gain is small in absolute terms, and it costs the largest jump in memory.

Q5_K_S against Q5_K_M is a smaller question: 0.12 GiB apart for an 8B model, with the M mix measuring about a third less divergence. Take M unless those megabytes decide whether the model fits.

IQ quants, imatrix and dynamic quants

Importance matrix. llama.cpp’s llama-imatrix tool runs a calibration text through the model and records which weights influence the output most. llama-quantize --imatrix then spends its precision there. The file name rarely shows whether one was used; look for “imatrix” or “i1” in the repository name or card.

IQ quants. The i-quants were added after the k-quants, for bit rates where k-quants fall apart. In llama.cpp’s table they run from IQ1_S at 2.00 bits per weight to IQ4_NL at 4.68. Their value is quality per gigabyte: in the Llama 3 8B scoreboard, IQ4_XS at 4.14 GiB shows a KL divergence of 0.036, which sits between Q4_K_S with an importance matrix (0.032) and without one (0.043), in a file 5% smaller than either.

Dynamic quants. Unsloth’s UD- files go a step further. Its model cards describe choosing each layer’s type from a measured sensitivity scan, and its documentation describes its own calibration data and calls the method post-training quantization only. The current generation is Dynamic 3.0, introduced with Qwen3.8-27B, and a 3.1 update is dated November 2026. Unsloth claims more than 10% better top-1 accuracy than other publishers’ files at the same size; that is the publisher’s claim, measured on its own benchmark. A UD-Q4_K_XL file loads in llama.cpp like any other GGUF; the name describes the recipe, not a new format. Our comparison of Unsloth Studio and LM Studio covers the tools around those files.

Other mixed-precision recipes exist. IST-DASLab publishes “GSQ-RCO” GGUF builds that it describes as dynamic non-uniform quantization, and Meta ships its own “K-Quant” builds of Muse Glimmer.

Newer types in llama.cpp. The quantizer’s type list on 14 November 2026 also includes Q1_0 (1.125 bits per weight), Q2_0 (2.25), the ternary TQ1_0 and TQ2_0, and MXFP4_MOE for mixture-of-experts models. These are specialised. If you are choosing a file for a standard model, the k-quants and i-quants above are still the ones on offer.

To make and measure a quant yourself:

./build/bin/llama-imatrix -m model-bf16.gguf -f calibration.txt -o imatrix.gguf
./build/bin/llama-quantize --imatrix imatrix.gguf model-bf16.gguf model-Q4_K_M.gguf Q4_K_M

llama.cpp’s perplexity tool then compares the result with the 16-bit model through its --kl-divergence-base and --kl-divergence options.

Which GGUF quantization should I use?

Work from memory first, then task. The picks below are ours, drawn from the measurements above.

On tool calling, the evidence is thinner than for perplexity, because few publishers measure it by level. The direction is consistent, though. Unsloth’s documentation says 1-bit files should not be used for agentic work, that heavily quantized files should not be used for tool calling, and that files below its 2-bit UD-Q2_K_XL can loop. If an agent misbehaves at Q4, try the same model at Q5_K_M or Q6_K before you blame the model. Our guides to local models for agents and tool calling and to local models for coding name models that hold up.

Two defaults are worth knowing. llama.cpp’s -hf download flag takes user/model:quant and falls back to Q4_K_M when you give no level. In the Ollama library, the default llama3.1 tag is the Q4_K_M build at 4.9 GB, and higher levels have explicit tags:

# llama.cpp: pick the level after the colon
llama-server -hf lmstudio-community/Qwen3.8-27B-GGUF:Q6_K -c 16384 -ngl all

# Ollama: the default tag is Q4_K_M; ask for another by name
ollama run llama3.1:8b-instruct-q8_0

For which model to put on a given card in the first place, use our model picks by VRAM tier.

Context memory on top of the weights

The file size is not the whole memory bill. Every token in the context window also occupies the key-value cache, and weight quantization does not shrink it.

For a standard transformer the cache per token is:

2 × layers × KV heads × head dimension × bytes per value

Llama 3.1 8B has 32 layers, 8 key-value heads and a head dimension of 128. With llama.cpp’s default 16-bit cache that is 2 × 32 × 8 × 128 × 2 = 131,072 bytes per token, by our arithmetic:

So an 8B model at Q4_K_M with its full 128K context needs more memory than the same model at Q8_0 with 32K. If you are short, a smaller context often buys more than a lower quantization level. llama.cpp can also quantize the cache itself with --cache-type-k and --cache-type-v, which accept types such as q8_0; the default is f16. Add compute buffers and, for vision models, the projector on top.

Architectures differ. Models with fewer key-value heads, sliding-window attention or linear-attention layers keep less per token, so calculate for your model, not from this table.

GGUF vs MLX vs safetensors, AWQ, GPTQ and EXL3

GGUF is one of several ways to ship a quantized model. Which you want depends on the runtime and the hardware.

Sizes for one model make the comparison concrete. For Qwen3.8-27B on 14 November 2026: GGUF Q4_K_M 16.81 GB and Q8_0 29.05 GB; MLX 4-bit 16.08 GB and 8-bit 29.53 GB from LM Studio’s community builds; Qwen’s own FP8 safetensors 30.89 GB; and a community AWQ INT4 build 21.04 GB.

GGUF vs MLX on a Mac. The files are about the same size at the same bit width, and LM Studio runs both, so the choice is between runtimes, not formats. We have no sourced speed comparison to quote for current versions, so run both on your own machine with the same prompt and context. MLX files do nothing for you on Windows or Linux, and GGUF is the format every local tool reads. Our guide to local models on Macs covers memory planning there.

GGUF vs server formats. On a GPU server with many concurrent users, the usual choice is safetensors in FP8 or a 4-bit AWQ or GPTQ build on vLLM or SGLang, not GGUF. Our comparison of vLLM, Ollama and llama.cpp explains why the serving engine matters more than the file format at that scale.

How VDF AI fits

VDF AI does not produce, host or distribute GGUF files, and it does not run, resell or integrate llama.cpp, LM Studio, Unsloth or the other formats and runtimes named on this page. The one exception is an Ollama server you run yourself, which VDF AI Router can call as an endpoint.

The link to an organisation is this: a Q4_K_M build and a Q8_0 build of one model behave differently, so the quantization level belongs in the record of what was approved, alongside the model name and licence. One person can keep that in their head. A team needs it written down, with access control over who may use which endpoint and logs of what was asked. VDF AI Router is the piece that sits in front of those endpoints: it registers Ollama and custom on-premises deployments next to any cloud models policy permits, and in air-gap mode it routes to the local ones alone.