Best Local LLM by VRAM: 8, 12, 16, 24 and 32 GB (2026)

The best local LLM for your VRAM is the strongest open-weight model whose 4-bit file plus KV cache fits in about 90% of the card. In October 2026 that means Qwen3.5-9B or Gemma 4 E4B on 8 GB, Gemma 4 12B on 12 GB, gpt-oss-20b on 16 GB, Qwen3.6-35B-A3B or Gemma 4 26B A4B on 24 GB, and Qwen3.6-35B-A3B with its full 256K window on a 32 GB RTX 5090.

Most “which model fits my GPU” answers stop at the weights. That is half the job. Every token you keep in the conversation also occupies memory, and two models of the same size can differ fivefold in how much. This guide picks models tier by tier and shows the context each one leaves you, so you can tell a model that loads from a model you can actually work with.

Every model, licence and parameter count below was checked on the vendor’s Hugging Face card, and every file size on the published GGUF repository, on 6 October 2026. GPU memory figures come from NVIDIA’s comparison page.

Quick picks by VRAM tier

Context figures assume one session and a 16-bit KV cache. If you write code with these models, our local coding model comparison covers coder-specific checkpoints.

How the fit is calculated

The method is the one behind our VRAM and KV cache calculator and the GPU sizing guide:

1. Usable memory is 90% of the card, leaving room for the runtime, activations and the desktop.
2. Weights are the size of the file you actually load. We use the GGUF published by the llama.cpp project’s ggml-org account, or by the model vendor where it ships its own (Mistral and IBM), or LM Studio’s community build for the small Qwen3.5 models.
3. KV cache per token is 2 × attention layers × KV heads × head dimension × 2 bytes at 16-bit. Sliding-window layers stop growing at their window. Hybrid models such as Qwen3.5 and Nemotron also hold a small fixed state per session, which we add separately.
4. Context that fits is the memory left after steps 2 and 3, divided by the per-token cost, capped at the model’s own limit.

Real files rarely match the textbook arithmetic. Embeddings, output layers and vision encoders are often kept at higher precision, and different builders make different choices. Qwen3.6-27B at Q4_K_M is 19.1 GB from ggml-org but 17 GB in Ollama’s library, so check the file you download. The tables assume text only; loading a vision projector adds 0.6 to 1.2 GB.

Worked example: Gemma 4 31B on a 24 GB card

• Usable: 24 × 0.9 = 21.6 GB.
• Weights: the Q4_0 file is 17.99 GB.
• Sliding-window layers: 50 layers × 2 × 16 KV heads × 256 × 2 bytes = 819,200 bytes per token, but only for the last 1,024 tokens, so a fixed 0.84 GB.
• Global layers: 10 × 2 × 4 KV heads × 512 × 2 bytes = 81,920 bytes per token, for the whole conversation.
• Room for context: (21.6 − 17.99 − 0.84) GB ÷ 81,920 bytes ≈ 33,800 tokens. An 8-bit KV cache stretches that to about 77,900.

The calculator’s Gemma 4 31B preset at GGUF Q4_K_M (4.89 bits per weight, 19.1 GB) returns about 20,000 tokens for the same card. The gap is the file: Q4_K_M stores more bits per weight than this Q4_0 build, and that difference comes straight out of your context.

Best LLM for 8 GB VRAM

Cards in this tier include the RTX 5060, the 8 GB RTX 5060 Ti, the RTX 4060 and 4060 Ti 8 GB, and the RTX 3070. Usable memory is about 7.2 GB.

The spread in the context column is architecture, not size. Qwen3.5 keeps a growing cache in only 8 of its 32 layers, 32,768 bytes per token, while Granite 4.2 8B caches all 40 layers at 163,840 bytes per token. Gemma 4 E4B’s figure is conservative: it assumes every layer keeps its own cache, although the last 18 layers are designed to share earlier ones. Qwen3.5 models think before answering by default, so long reasoning traces eat into the same budget.

Best LLM for 12 GB VRAM

Examples are the RTX 5070, the RTX 4070, 4070 SUPER and 4070 Ti, the 12 GB RTX 3060 and the RTX 3080 Ti. Usable memory is about 10.8 GB.

Gemma 4 12B is the standout. Only 8 of its 48 layers attend over the whole conversation, each with a single 512-dimension KV head, so a 200K-token chat costs about 3.6 GB of cache, sliding-window layers included. gpt-oss-20b does not fit: its 12.1 GB file exceeds the 10.8 GB budget. llama.cpp can keep some mixture-of-experts weights in system RAM with --n-cpu-moe, at a cost in speed.

Best LLM for 16 GB VRAM

This is the most common enthusiast and workstation tier: RTX 5080, RTX 5070 Ti, the 16 GB RTX 5060 Ti, RTX 4080 and 4080 SUPER, RTX 4070 Ti SUPER and the 16 GB RTX 4060 Ti. Usable memory is about 14.4 GB.

gpt-oss-20b was trained with MXFP4 weights, so there is no separate quantization step to second-guess; OpenAI’s card says it runs within 16 GB. OpenAI says the model must be used with its harmony response format; the chat template in the repository applies it, and llama.cpp lists a native handler for it.

What does not fit: Mistral Small 3.2 24B at Q4_K_M is 14.3 GB and leaves a few hundred tokens, Gemma 4 26B A4B’s 14.6 GB Q4_0 file is just over budget, and the 27B Qwen models need 19 GB. One trap applies to this whole tier: Ollama’s documentation says it defaults to a 4K context on GPUs with less than 24 GiB, so set OLLAMA_CONTEXT_LENGTH to the window you sized for.

Best LLM for 24 GB VRAM

The RTX 4090, RTX 3090 and RTX 3090 Ti sit here. Usable memory is about 21.6 GB, and the choice widens to 24B–36B models.

Two notes on the dense Qwen models. Qwen’s Qwen3.6 card advises keeping at least 128K of context to preserve thinking quality, which a 24 GB card cannot give the 27B at 16-bit. An 8-bit KV cache roughly doubles the figure; turning thinking off for simple requests is the other lever. Qwen3.6-27B behaves like Qwen3.8-27B here, with a 19.1 GB file and about 36K tokens.

Best LLM for 32 GB VRAM (RTX 5090)

The GeForce RTX 5090 is the only 32 GB card in NVIDIA’s consumer line. NVIDIA lists 32 GB of GDDR7 on a 512-bit interface, 1,792 GB/s of memory bandwidth, 575 W of total graphics power and a 1,000 W system power requirement (verified October 2026). Usable memory is about 28.8 GB.

The extra 8 GB over a 24 GB card mostly buys context and concurrency rather than bigger models. 8-bit 27B files (28.6 GB) load but leave almost nothing for the cache, and 70B dense models are out of reach: Ollama’s Llama 3.3 70B at q4_K_M is 43 GB. For that class, look at 48 GB-plus GPUs or the unified-memory machines in our DGX Spark, Mac Studio and Ryzen AI Max comparison.

When the model you want does not fit

Five levers, in the order we would pull them:

1. Quantize the KV cache. llama.cpp accepts --cache-type-k q8_0 (and the matching V flag), Ollama exposes OLLAMA_KV_CACHE_TYPE=q8_0, and vLLM supports an FP8 KV cache. Ollama’s FAQ describes q8_0 as roughly half the memory of f16 with a very small loss in precision.
2. Shorten the window. Size context for the task, not the model card’s maximum.
3. Offload experts. For mixture-of-experts models, llama.cpp’s --n-cpu-moe N keeps the expert weights of the first N layers in system RAM. It runs, slower.
4. Let llama.cpp fit it. Current llama-server builds adjust unset arguments to device memory by default (--fit), keeping a 1,024 MiB margin per device unless you change it.
5. Drop a quantization level. Weight precision is the last lever, because quality loss varies by model and task. Our quantization trade-off guide explains how to test it.

If several people will share one GPU, every concurrent session needs its own cache, and single-user runners queue requests by default. Our vLLM, Ollama and llama.cpp comparison covers the batching servers built for that. For the same picks on Apple hardware, see the Mac memory-tier guide; for retrieval and agent workloads, see our picks for local RAG and tool calling. To pick the card itself, the GPU buyer’s guide compares them by budget, and the Gemma 4 setup guide walks through one family end to end.

How VDF AI fits

A single GPU under a desk serves one person well. When a team needs the same local models, VDF AI Router registers Ollama and custom on-premises deployments alongside any cloud models your policy allows, and its air-gap mode restricts routing to local models only. Local runtimes are probed continuously, so a request can move to another permitted model when one endpoint is down.

That lets you keep a small model on modest hardware for routine requests and send heavier work to a larger model on a server, behind one API. The local LLM overview covers the wider deployment picture.