llama-server is the HTTP server that ships with llama.cpp. Start it with llama-server -hf followed by a Hugging Face repository, or -m and a GGUF file, and it serves OpenAI-compatible and Anthropic-compatible APIs plus a web chat UI. Since build b11521 of 9 October 2026 it listens on 127.0.0.1:9931 by default, not 8080, and it has no authentication unless you set --api-key.
Most problems with the llama.cpp server come from three places: a tutorial written for an older build, a default that changed, or a flag that does something different from what its name suggests. This guide walks through the server as it is now: installing it, starting a model, the flags worth knowing, the API, the web UI, router mode, tool calling, MCP, running it as a service and exposing it safely.
Commands and defaults come from the llama.cpp repository’s server README, install and Docker docs, function-calling doc and security policy, read on 14 November 2026 against build b12387, published that day. llama.cpp publishes several builds a day, so run llama-server --help on your own build before trusting any default, ours included. If you are still choosing an engine, our inference server comparison sets llama.cpp against the alternatives.
What llama-server is
llama.cpp is an MIT-licensed inference engine written in C and C++ for models in the GGUF format, with about 135,000 GitHub stars at the time of writing. llama-server is its HTTP front end. The README lists what it provides:
- OpenAI-compatible chat completions, responses and embeddings routes.
- An Anthropic Messages-compatible route.
- A reranking endpoint and monitoring endpoints.
- Parallel decoding with multi-user support and continuous batching.
- Multimodal input, schema-constrained JSON output, function calling and speculative decoding.
- A web UI served from the same port.
The project has also added a unified command, llama, with subcommands: the main README now shows llama serve -hf ... to launch the API server and llama cli -hf ... for a terminal chat. A menu-bar and system-tray app called Llama, from the llama.cpp team and Hugging Face, wraps the same server for Mac and Windows. This guide uses the llama-server binary, which every packaged build still contains.
Install llama-server on Windows, macOS and Linux
| Method | Command or source | Systems |
|---|---|---|
| Install script | curl -LsSf https://llama.app/install.sh | sh | Linux, macOS |
| Install script (PowerShell) | irm https://llama.app/install.ps1 | iex | Windows |
| Winget | winget install llama.cpp | Windows |
| Homebrew | brew install llama.cpp | macOS, Linux |
| conda-forge | conda install -c conda-forge llama.cpp | Windows, macOS, Linux |
| MacPorts, Nix | sudo port install llama.cpp, nix profile install nixpkgs#llama-cpp | macOS; macOS and Linux |
| Prebuilt binaries | The GitHub releases page | Windows, macOS, Linux, Android |
| Docker | ghcr.io/ggml-org/llama.cpp:server and GPU variants | Linux hosts |
| Source | cmake -B build && cmake --build build --config Release -t llama-server | Any |
The release page has one archive per back end. For build b12387 the names include llama-b12387-bin-macos-arm64.tar.gz, llama-b12387-bin-ubuntu-x64.tar.gz, llama-b12387-bin-ubuntu-vulkan-x64.tar.gz and llama-b12387-bin-ubuntu-cuda-13.5-x64.tar.gz, with ROCm, SYCL and OpenVINO builds beside them. Pick the archive that matches your GPU; the plain Ubuntu x64 and arm64 archives are CPU-only.
llama.cpp server on Windows
Winget is the shortest path and is updated automatically with new releases. If you prefer the archives, download the build for your GPU, such as llama-b12387-bin-win-cuda-13.5-x64.zip for NVIDIA or llama-b12387-bin-win-vulkan-x64.zip for AMD and Intel, and for CUDA also the matching cudart-llama-bin-win-cuda-13.5-x64.zip, which carries the CUDA runtime libraries. Unzip both into one folder and run:
.\llama-server.exe -m models\my-model.gguf -c 8192
There are also Windows on ARM builds, including one for Adreno GPUs.
Docker
The server images are server for CPU and server-cuda, server-cuda13, server-vulkan, server-rocm and server-intel for GPUs. The documented command still maps port 8080, because the images keep that port inside the container:
docker run -p 8080:8080 -v /path/to/models:/models --gpus all \
ghcr.io/ggml-org/llama.cpp:server-cuda \
-m /models/my-model.gguf --host 0.0.0.0 --port 8080 --n-gpu-layers 99
Start a model with -hf or -m
To download a model from Hugging Face and serve it:
llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M
The tag after the colon chooses the quantisation. It is optional and defaults to Q4_K_M, or the first file in the repository if no Q4_K_M file exists. If the repository includes a multimodal projector, it is downloaded too; add --no-mmproj to skip it. A gated repository needs -hft or the HF_TOKEN environment variable.
To serve a file you already have:
llama-server -m ./models/my-model.gguf -c 16384 --alias my-model
Then test it from another terminal:
curl http://localhost:9931/health
curl http://localhost:9931/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "my-model", "messages": [{"role": "user", "content": "Hello"}]}'
/health returns 503 while the model loads and {"status": "ok"} once it is ready. Two flags help on restricted networks: --offline forces the cache and prevents network access, and HTTPS_PROXY is honoured for model downloads behind a corporate proxy.
Not sure a model will fit? Our VRAM calculator estimates memory for a model, quantisation and context length, and the quantisation guide explains what Q4_K_M trades away.
llama.cpp server flags that matter
The README’s flag list runs to well over 200 rows. These are the ones that decide whether a set-up works.
| Flag | Default | What it does |
|---|---|---|
-m, --model | none | Path to a GGUF file |
-hf, --hf-repo | none | Hugging Face repository, with optional :quant |
-c, --ctx-size | 0 | Context size; 0 means read it from the model |
-fit, --fit | on | Adjusts unset arguments to fit device memory; --fit-ctx sets the smallest context it may pick (4096) |
-ngl, --gpu-layers | auto | Layers to keep in VRAM: a number, auto or all |
-np, --parallel | -1 (auto) | Number of server slots, meaning requests handled at once |
--host | 127.0.0.1 | Listen address; accepts several addresses or a UNIX socket path |
--port | 9931 | Listen port |
--api-key, --api-key-file | none | Require a key; several keys allowed |
-a, --alias | none | Model name or names reported to API clients |
--jinja | enabled | Use the Jinja chat template engine, which tool calling needs |
-fa, --flash-attn | auto | Flash Attention on, off or auto |
-to, --timeout | 3600 | Read and write timeout in seconds |
--metrics | disabled | Prometheus-compatible /metrics endpoint |
--slots | enabled | Slot monitoring endpoint |
--sleep-idle-seconds | -1 (disabled) | Unload the model after this much idle time |
--embedding, --rerank | disabled | Serve an embedding model, or enable the reranking endpoint |
Three behaviours catch people out.
Context and --fit. With -c unset, the server reads the context length from the model, and --fit, which is on by default, adjusts unset arguments so the load fits in device memory. The README’s router section says an instance then takes the largest context that fits at load time. That is convenient on one machine and unpredictable across machines, so set -c explicitly for anything you share or script.
Context and slots. When you set -np yourself, the context is shared out between slots. Hermes Agent’s documentation gives the standard example: -c 64000 -np 4 leaves 16K tokens per slot. With slots left on auto, the server uses a single unified KV buffer shared across sequences, and --kv-unified-per-slot caps what one slot may use. For a coding agent that needs 64K, either run one slot or multiply the context.
Environment variables. Most flags have an environment form, such as LLAMA_ARG_CTX_SIZE, LLAMA_ARG_N_GPU_LAYERS, LLAMA_ARG_PORT and LLAMA_API_KEY. A command-line argument wins when both are set.
The default port changed from 8080 to 9931
For years llama-server listened on port 8080. The project added a notice about an upcoming change on 3 August 2026 (pull request 26508) and switched the default on 9 October 2026 (pull request 30159, first shipped in build b11521). The README now says the server listens on 127.0.0.1:9931 by default.
What this breaks, and how to fix it:
- Client configs and tutorials that assume 8080. Many coding agents document a llama.cpp provider on port 8080. Pass
--port 8080to the server, or change the client’s base URL tohttp://127.0.0.1:9931/v1. - Docker. Nothing changes: the pull request kept 8080 inside the container so tools built on the base image keep working, and the documented commands pass
--port 8080explicitly. - Auto-detection. Tools that probe 8080 for a running server will not find one on the new default. Hermes Desktop is one; our Hermes Desktop set-up guide shows the config line that points it at 9931.
The safest habit is to write the port into every start command and unit file. Our set-up guides for OpenCode, Claude Code and the Pi coding agent each show the client side for their tool.
llama.cpp server API: OpenAI and Anthropic endpoints
| Endpoint | Purpose |
|---|---|
GET /health | Readiness check; public even when an API key is set |
GET /v1/models | The loaded model, with input and output modalities |
POST /v1/chat/completions | OpenAI-compatible chat, streaming, tools, JSON schema output |
POST /v1/completions | OpenAI-compatible text completion |
POST /v1/responses | OpenAI-compatible Responses API |
POST /v1/embeddings | OpenAI-compatible embeddings |
POST /v1/messages | Anthropic-compatible Messages API, with /v1/messages/count_tokens |
POST /reranking | Rerank documents against a query (needs --rerank) |
POST /completion, /tokenize, /detokenize, /infill | Native llama.cpp endpoints |
GET /props, /slots, /metrics | Server properties, slot state, Prometheus metrics |
The README is careful about compatibility: it makes no strong claim to match the OpenAI or Anthropic specifications, only that the routes work with many applications. In practice, point an OpenAI client at the /v1 base URL:
import openai
client = openai.OpenAI(
base_url="http://localhost:9931/v1",
api_key="sk-no-key-required",
)
For Anthropic-style clients, the Messages route accepts system, tools, tool_choice and streaming, and the README’s example sends the key in an x-api-key header. Fields llama.cpp has but the vendor APIs lack, such as seed or min_p, go in an extra_params object on the Messages and Responses routes.
Schema-constrained output works through response_format on chat completions, with either json_object or a JSON schema. Recent builds also list a WebSocket /v1/realtime route for transcription and conversation and an OpenAI-compatible speech route.
The built-in web UI
Open http://localhost:9931 in a browser and you get a chat interface served by the same process. It is enabled by default; --no-ui turns it off for API-only deployments. Defaults for the UI can be set with --ui-config or --ui-config-file, and the older --webui- spellings still work as aliases.
The UI is also where the server’s agent features surface. With --tools, the model can call built-in tools from the UI: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file and get_info. The --agent flag enables all of them plus a CORS proxy. The README marks these as experimental and says not to enable them in untrusted environments, because they give the model read and write access to the host’s files and a shell. --tools-runtime can run them in a Docker or Podman container or over SSH instead of on the host.
Router mode: several models on one server
Start llama-server with no model and it runs as a router: a parent process that starts a child server per model and forwards requests.
llama-server --models-dir ./models --models-max 2
Models can come from three places: the download cache, a directory given with --models-dir, or an INI file given with --models-preset. A preset file sets per-model arguments:
version = 1
[*]
c = 8192
[coder]
model = /srv/models/coder-Q4_K_M.gguf
n-gpu-layers = all
c = 65536
Requests are routed by the model field in the JSON body, or a model query parameter on GET endpoints. Models load on first use unless you pass --no-models-autoload. --models-max limits how many stay loaded (default 4), unloading the least recently used, and --models-memory-margin makes the router measure each model and free memory before loading another. GET /models lists models and their state, and POST /models/load and /models/unload control them.
Router mode is useful on a workstation that switches between a chat model, a coding model and an embedding model. Be aware of its status, though: the project’s security policy lists the router among features not recommended for untrusted environments, and an issue opened on 11 October 2026 asks for authentication between the router and its instances.
Tool calling and Jinja templates
Tool calling depends on the model’s chat template. The server’s Jinja template engine is enabled by default in current builds (--jinja, with --no-jinja to switch it off), so the old advice to add --jinja is now a precaution, not a requirement. The Anthropic route’s documentation still words it as needing --jinja.
The function-calling doc describes two paths:
- Native handlers for formats the server recognises, including Llama 3.x, Functionary, Hermes 2 and 3, Qwen 2.5 and Qwen 2.5 Coder, Mistral Nemo, Firefunction v2, Command R7B and GPT-OSS.
- A generic handler for everything else. It works, but the doc says it may use more tokens and be less efficient. The log line
Chat format: Generictells you it is in use.
If a model’s embedded template is wrong or missing tool support, override it with --chat-template-file. To check what the server loaded, look at the chat_template property at /props. Parallel tool calls are off by default and enabled per request with "parallel_tool_calls": true. For reasoning models, --reasoning-format controls whether thinking is returned in reasoning_content, and --reasoning-budget caps it.
Template support matters as much as model size here. Our guide to local models for tool calling lists models that handle agent loops reliably.
External tool servers in llama-server
llama-server can act as an MCP host. Pull request 26062, merged on 25 July 2026, added support for MCP servers over the stdio transport. Declare them in a Cursor-compatible JSON file:
{
"mcpServers": {
"files": { "command": "/usr/local/bin/my-mcp-server", "args": ["--root", "/srv/docs"] }
}
}
llama-server -m model.gguf --mcp-servers-config mcp.json
Each entry accepts command, args, env, cwd and timeout_ms (30 seconds by default). The server starts each MCP server once at launch to list its tools, stops it, and starts it again when a tool is called. Tools appear to the model and the web UI as <server>_<tool>.
The limits are stated plainly in the README:
- Stdio only. Remote HTTP MCP servers are not supported by this feature. The separate
--ui-mcp-proxyflag only lets the web UI reach public MCP servers that lack CORS headers, and it refuses local and private addresses. - Same privileges. An MCP server runs as a child process with llama-server’s own permissions, so declare only commands you trust.
- Experimental. The flag is labelled experimental, and enabling it narrows
--cors-originsto localhost by default. Tools and MCP on a non-loopback address need an API key unless you pass--allow-unauthenticated-tools.
Note the direction: this makes llama-server a client of MCP servers. It does not turn llama-server into an MCP server that other agents connect to. The web UI can also connect to MCP servers by URL from the browser; our guide to MCP with local model runtimes covers that side and compares it with LM Studio and Open WebUI.
Run llama-server as a service
llama.cpp does not ship a service installer, so use the host’s own tools. On Linux, a minimal systemd unit looks like this; it is our example, not an upstream file:
[Unit]
Description=llama-server
After=network-online.target
[Service]
User=llama
ExecStart=/usr/local/bin/llama-server -m /srv/models/model.gguf -c 32768 --host 127.0.0.1 --port 9931 --api-key-file /etc/llama/keys --metrics
Restart=on-failure
[Install]
WantedBy=multi-user.target
With Docker Compose, the README’s pattern is to configure the server through environment variables:
services:
llamacpp-server:
image: ghcr.io/ggml-org/llama.cpp:server
ports:
- 8080:8080
volumes:
- ./models:/models
environment:
LLAMA_ARG_MODEL: /models/my_model.gguf
LLAMA_ARG_CTX_SIZE: 4096
LLAMA_ARG_N_PARALLEL: 2
LLAMA_ARG_ENDPOINT_METRICS: 1
LLAMA_ARG_PORT: 8080
For a server that sits idle most of the day, --sleep-idle-seconds unloads the model and its cache after the given idle time and reloads on the next request. Health, props, models and metrics calls do not wake it. --metrics exposes token throughput, busy slots and deferred requests for Prometheus, and --log-file writes logs to a file.
Security when exposing llama-server
The defaults are safe for one machine and nothing more: the server binds to 127.0.0.1, has no API key, and reflects any Origin header for CORS. Before changing --host, work through this list.
- Set an API key. Use
--api-key-fileso keys stay out of the process list. Remember that/healthstays public. - Add TLS. A build with
-DLLAMA_OPENSSL=ONaccepts--ssl-key-fileand--ssl-cert-file. Most teams terminate TLS at a reverse proxy instead. - Put a reverse proxy in front. The README’s CORS table recommends an API key plus a reverse proxy for public deployments, and a specific
--cors-originsvalue on a local network. - Keep agent features off. Do not combine
--tools,--agentor MCP with a network-reachable address unless every client is trusted. - Leave
--connectalone. Added on 19 October 2026 (pull request 28277), it opens an experimental peer-to-peer tunnel to the server from outside the LAN. - Treat models as untrusted input. The security policy advises running models from unknown sources in a container or VM and checking the hash of every downloaded file.
- Know what the project covers. The policy excludes the web UI, experimental features, the router, MCP and most denial-of-service bugs from its vulnerability scope, and says the private disclosure programme is currently disabled.
An API key shared by everyone is not user management. There are no accounts, quotas per person or audit records in llama-server, and its own policy puts tenant isolation on the operator.
Where the llama.cpp server docs live
There is no separate documentation site for the server. The reference is in the repository:
tools/server/README.md: every flag, generated from the binary’s help text, and every endpoint.docs/function-calling.md: tool calling, templates and examples.docs/install.md,docs/docker.mdanddocs/build.md: packages, images and compiling.SECURITY.md: the security policy.- Issue 9291: the server’s REST API changelog.
Because the flag list is generated from the code, llama-server --help on your own build is the most accurate reference for that build.
How VDF AI fits
llama-server is a model server: it turns a GGUF file into an API on one machine. A team that wants many people and agents to use private models needs things it leaves out by design, such as sign-in, access rules per group, approved-model lists and a record of who asked what. VDF AI does not ship, host, resell or support llama.cpp.
VDF AI is an on-premises and private-cloud platform for governed agents, multi-agent Networks and private RAG, with role-based access control and audit logs on every plan. VDF AI Router registers Ollama and custom on-premises deployments next to the cloud models your policy approves, and its air-gap mode keeps routing on local models only. VDF AI documents no llama-server integration, so treat that pairing as something to test in a pilot.
Sources
Repository files, pull requests and release pages read on 14 November 2026 (latest build that day: b12387).
- llama.cpp: repository, server README, function calling, install, Docker and security policy
- Port change: pull request 30159, pull request 26508 and build b11521
- MCP: pull request 26062; peer-to-peer tunnel: pull request 28277
- Router authentication: issue 30349
- llama.cpp releases and the Llama app site
- Hermes Agent documentation on slots and context: providers
Moving from one model server to private AI for a whole team? See how VDF AI Router applies model policy, or book a demo.