vLLM vs Ollama vs llama.cpp vs SGLang: Choosing an LLM Inference Server in 2026

Every project in this comparison loads open-weight models and answers on an OpenAI-style HTTP API, so from an application’s side they look interchangeable. The differences appear when fifty people send requests at the same moment, when a model is too big for one GPU, or when the server has to run on AMD, Apple or CPU-only hardware. Facts below come from each project’s documentation and repository, checked on 2 October 2026. All of them release often, so confirm version-specific details before you standardise.

For how each one sits inside a larger architecture, see where vLLM fits and where Ollama fits. This post is the head-to-head.

The options at a glance

The latest releases on 2 October 2026 were vLLM v0.30.0 (22 September), SGLang v0.5.21 (2 October), llama.cpp v0.5.0 (23 September) and Ollama v0.35.0 (28 September). NVIDIA’s own stack (Dynamo-Triton, TensorRT-LLM and Dynamo) is covered in its own section below, because it plays a different role.

vLLM vs Ollama

Concurrency is the deciding difference. vLLM was built around PagedAttention, which manages the key-value cache in pages, and continuous batching, which adds new requests to a running batch instead of waiting for it to finish (vLLM README). Ollama’s FAQ sets OLLAMA_NUM_PARALLEL to 1 by default, so each loaded model handles one request at a time while up to 512 wait in the queue (Ollama FAQ). You can raise that number, but each parallel slot reserves its own context memory.

A named benchmark shows the size of the gap. Red Hat tested both on one A100 40 GB GPU with Llama 3.1 8B at 16-bit, vLLM 0.9.1 against Ollama 0.9.2, from 1 to 256 concurrent users. vLLM peaked at 793 tokens per second against 41 for Ollama, which had been tuned to 32 parallel requests, with a 99th-percentile latency of 80 ms against 673 ms (Red Hat, August 2025). Red Hat promotes its own vLLM-based product on the same page, and both projects have shipped many releases since, so treat the result as an order of magnitude rather than a current figure.

Formats follow from the purpose. vLLM serves Hugging Face checkpoints in 16-bit, FP8, MXFP4, NVFP4, INT8, INT4, GPTQ and AWQ, and its documentation calls GGUF support “highly experimental and under-optimized”. Ollama runs GGUF files and imports Safetensors weights through a Modelfile (import guide). If your model only exists as a GGUF file, that alone points to Ollama or llama.cpp.

Operations differ too. vLLM claims 92% of each GPU’s memory by default (--gpu-memory-utilization), which suits a dedicated server and not a shared workstation, and it collects anonymous usage data unless you set VLLM_NO_USAGE_STATS=1. Ollama keeps up to three models per GPU loaded at once by default. Its local API has no authentication, so anything beyond a single machine needs a proxy in front.

The rule of thumb: Ollama for one person or a small pilot, vLLM once a GPU server is shared by a team or an application.

Ollama vs llama.cpp

The two are related. Ollama’s May 2025 engine announcement says it had “so far relied on” the llama.cpp project for model support, and that accessing GGML directly from Go now lets it build its own inference graphs (Ollama blog). GGML is the tensor library underneath llama.cpp.

What llama.cpp gives you that Ollama hides:

• Backends. CUDA, HIP for AMD, Metal for Apple silicon, Vulkan, SYCL for Intel GPUs, CANN for Ascend, OpenCL for Adreno and plain CPU builds, among others (README).
• Server controls. llama-server exposes OpenAI-compatible chat completions, responses and embeddings, plus Anthropic Messages. It runs parallel slots (-np) with continuous batching on by default, splits models across GPUs by layer, row or tensor (-sm, -ts), and accepts an optional --api-key (server docs).
• Distributed experiments. An RPC backend spreads a model over several machines, but its own README calls it a proof of concept that is “fragile and insecure”.

What Ollama adds on top: a model library with one-command pulls, Modelfiles for packaging prompts and parameters, automatic placement across the available GPUs, and an API that many tools already support. It also runs models in Ollama’s cloud from the same apps and terminal, so a deployment that must keep prompts on your hardware should decide whether those models are allowed.

Governance has moved as well. The ggml.ai team that founded llama.cpp joined Hugging Face in February 2026, saying the project stays fully open source, and in September 2026 NVIDIA agreed to acquire Hugging Face (verified October 2026). Both projects remain MIT-licensed.

vLLM vs SGLang

These two are the production pair. Both are Apache-2.0, both serve OpenAI-compatible APIs, and both support tensor, pipeline, data and expert parallelism. SGLang is hosted by LMSYS, a non-profit; vLLM is a PyTorch Foundation project.

• Prefix reuse. SGLang’s RadixAttention reuses the cached work for prompt prefixes that requests share (SGLang docs). Agent loops, few-shot prompts and RAG pipelines with long common system prompts benefit most. vLLM also caches prefixes, so measure rather than assume.
• Hardware. vLLM lists NVIDIA, AMD and Intel GPUs and x86, ARM and PowerPC CPUs, with plugins for Google TPU, Intel Gaudi, Huawei Ascend and Apple silicon. SGLang lists NVIDIA, AMD, Intel Xeon, Google TPU, Ascend NPU and Moore Threads accelerators.
• APIs. vLLM also implements the Anthropic Messages API and gRPC. SGLang starts with python3 -m sglang.launch_server and listens on port 30000 by default.
• Model support. New open models usually ship with recipes for both. Mistral’s card for Mistral Small 4 recommends vLLM and lists SGLang, llama.cpp and LM Studio as compatible.

On speed, the most quoted comparison is SGLang’s own July 2024 benchmark, which reported up to 3.1 times the throughput of vLLM 0.5.2 on Llama 3 70B. It was written by the SGLang team against versions more than two years old. A fair answer today needs your model, your prompts and both latest releases.

Ollama vs LM Studio

Both target one machine, so the choice is mostly about interface and terms.

• Interface. LM Studio is a desktop app with a model browser and chat window, plus llmster, a headless daemon that runs on a Linux server without a GUI, and lms, its command-line tool (LM Studio docs). Ollama is command-line and API first.
• Engines. LM Studio runs models with llama.cpp on macOS, Windows and Linux and, on Apple silicon, with Apple’s MLX framework. Ollama accelerates Macs through Apple’s Metal API.
• APIs. LM Studio serves OpenAI-compatible and Anthropic-compatible endpoints on localhost:1234. Ollama serves a subset of both on port 11434.
• Licence. Ollama is MIT open source. LM Studio’s app is closed source, though its CLI and SDKs are MIT. It has been free for home and work since July 2025, but its terms (version of 23 August 2026) limit use to personal and internal business purposes and prohibit service-bureau and software-as-a-service use.

If your team needs a shared chat interface rather than a desktop app, a self-hosted web front end is the usual next step; Open WebUI, LibreChat and AnythingLLM are compared separately.

Where NVIDIA Triton, Dynamo and TensorRT-LLM fit

Searches for “vLLM vs Triton” mix up layers. NVIDIA’s stack has three parts:

• Dynamo-Triton, formerly Triton Inference Server, is a BSD-3-Clause server for many model types. Its backends include TensorRT, ONNX Runtime, PyTorch, Python, TensorRT-LLM and vLLM, so vLLM can run inside it. Choose it when one server must host language models next to vision, speech or classic machine-learning models.
• TensorRT-LLM is an Apache-2.0 engine for NVIDIA GPUs only. trtllm-serve starts an OpenAI-compatible server, and its quantization matrix covers NVFP4 on Blackwell and Rubin GPUs, MXFP4, FP8, and 4-bit AWQ and GPTQ.
• Dynamo is an Apache-2.0 framework for distributed serving at data-centre scale. It runs vLLM, SGLang or TensorRT-LLM as engines, separates prefill and decode into independently scaled GPU pools, and routes requests by worker load and KV-cache overlap. Version 1.5.0 shipped on 21 September 2026.

“vLLM vs LiteLLM” is a category mix-up too. LiteLLM describes itself as an open-source AI gateway that calls more than 100 LLMs in the OpenAI format. It does not run models; its documentation includes a provider for vLLM servers, so it sits in front of one. That layer is covered under AI gateways.

Which server for which workload

For Kubernetes clusters, the guide to hosting LLMs on Kubernetes covers how vLLM is packaged and scaled there.

How to benchmark them on your own hardware

Published comparisons rarely match your model, your prompts or your GPUs. A fair test takes a day:

1. Hold the model constant. Serve the same checkpoint at the same precision on every server. A 4-bit GGUF file on Ollama against 16-bit weights on vLLM measures quantization, not the server.
2. Replay real prompts. Collect a few hundred prompts with your actual input and output lengths, including long retrieval prompts if you run RAG.
3. Drive every server through its OpenAI-style endpoint. vllm bench serve reports time to first token, inter-token latency, end-to-end latency and throughput, and its openai-chat backend sends requests to a chat-completions endpoint. Step --max-concurrency through 1, 4, 16 and 64.
4. Record what users feel. 95th-percentile time to first token, output tokens per second per stream, total tokens per second and peak GPU memory.
5. Find the knee. The concurrency at which p95 time to first token crosses your target is that server’s capacity on that GPU. Size from that number, not from peak throughput.
6. Re-run after upgrades. These projects ship often, and a new release can move the result.

To turn the measured capacity into a hardware plan, use the GPU sizing method.

How VDF AI fits

VDF AI works above the inference server. VDF AI Agents can use Ollama or any OpenAI-compatible endpoint as an agent’s model, and VDF AI Chat serves open-weight models through the same kind of endpoint, so each server compared here can sit underneath them.

VDF AI Router gives applications one endpoint in front of those deployments. It registers local runtimes next to any external models policy allows, probes them continuously and, if a local runtime stops answering, shifts traffic to permitted cloud models instead of failing. In air-gap mode it routes to local models only. Model choice then lives in the router rather than in application code, so moving a workload from Ollama on a workstation to vLLM on a GPU server does not require an application change.

Sources

• vLLM repository and README, GGUF support, parallelism and scaling, engine arguments, usage statistics and vllm bench serve
• PyTorch Foundation welcomes vLLM
• SGLang documentation and repository
• llama.cpp repository and server documentation
• Hugging Face: ggml joins Hugging Face
• NVIDIA to acquire Hugging Face
• Ollama FAQ, GPU support, OpenAI compatibility, importing models, Ollama Cloud and new engine announcement
• LM Studio documentation, LM Studio vs llmster vs lms, free for work and app terms
• NVIDIA Dynamo-Triton, TensorRT-LLM and Dynamo
• LiteLLM repository and vLLM provider docs
• Red Hat: Ollama vs vLLM benchmark (August 2025)
• LMSYS: SGLang Llama 3 benchmark (July 2024)
• Mistral Small 4 model card