AI Infrastructure

llama.cpp Server Guide: llama-server Flags, API, Port 9931, Router Mode and Security

A practical guide to llama-server, the HTTP server that ships with llama.cpp: how to install it on Windows, macOS and Linux, start a model with -hf or -m, which flags matter, the default port change from 8080 to 9931, the OpenAI- and Anthropic-compatible endpoints, the built-in web UI, router mode for several models, tool calling, MCP support, running it as a service and securing it. Checked against build b12387 of 14 November 2026.

llama-server is the HTTP server that ships with llama.cpp. Start it with llama-server -hf followed by a Hugging Face repository, or -m and a GGUF file, and it serves OpenAI-compatible and Anthropic-compatible APIs plus a web chat UI. Since build b11521 of 9 October 2026 it listens on 127.0.0.1:9931 by default, not 8080, and it has no authentication unless you set --api-key.

Most problems with the llama.cpp server come from three places: a tutorial written for an older build, a default that changed, or a flag that does something different from what its name suggests. This guide walks through the server as it is now: installing it, starting a model, the flags worth knowing, the API, the web UI, router mode, tool calling, MCP, running it as a service and exposing it safely.

Commands and defaults come from the llama.cpp repository’s server README, install and Docker docs, function-calling doc and security policy, read on 14 November 2026 against build b12387, published that day. llama.cpp publishes several builds a day, so run llama-server --help on your own build before trusting any default, ours included. If you are still choosing an engine, our inference server comparison sets llama.cpp against the alternatives.

What llama-server is

llama.cpp is an MIT-licensed inference engine written in C and C++ for models in the GGUF format, with about 135,000 GitHub stars at the time of writing. llama-server is its HTTP front end. The README lists what it provides:

  • OpenAI-compatible chat completions, responses and embeddings routes.
  • An Anthropic Messages-compatible route.
  • A reranking endpoint and monitoring endpoints.
  • Parallel decoding with multi-user support and continuous batching.
  • Multimodal input, schema-constrained JSON output, function calling and speculative decoding.
  • A web UI served from the same port.

The project has also added a unified command, llama, with subcommands: the main README now shows llama serve -hf ... to launch the API server and llama cli -hf ... for a terminal chat. A menu-bar and system-tray app called Llama, from the llama.cpp team and Hugging Face, wraps the same server for Mac and Windows. This guide uses the llama-server binary, which every packaged build still contains.

Install llama-server on Windows, macOS and Linux

MethodCommand or sourceSystems
Install scriptcurl -LsSf https://llama.app/install.sh | shLinux, macOS
Install script (PowerShell)irm https://llama.app/install.ps1 | iexWindows
Wingetwinget install llama.cppWindows
Homebrewbrew install llama.cppmacOS, Linux
conda-forgeconda install -c conda-forge llama.cppWindows, macOS, Linux
MacPorts, Nixsudo port install llama.cpp, nix profile install nixpkgs#llama-cppmacOS; macOS and Linux
Prebuilt binariesThe GitHub releases pageWindows, macOS, Linux, Android
Dockerghcr.io/ggml-org/llama.cpp:server and GPU variantsLinux hosts
Sourcecmake -B build && cmake --build build --config Release -t llama-serverAny

The release page has one archive per back end. For build b12387 the names include llama-b12387-bin-macos-arm64.tar.gz, llama-b12387-bin-ubuntu-x64.tar.gz, llama-b12387-bin-ubuntu-vulkan-x64.tar.gz and llama-b12387-bin-ubuntu-cuda-13.5-x64.tar.gz, with ROCm, SYCL and OpenVINO builds beside them. Pick the archive that matches your GPU; the plain Ubuntu x64 and arm64 archives are CPU-only.

llama.cpp server on Windows

Winget is the shortest path and is updated automatically with new releases. If you prefer the archives, download the build for your GPU, such as llama-b12387-bin-win-cuda-13.5-x64.zip for NVIDIA or llama-b12387-bin-win-vulkan-x64.zip for AMD and Intel, and for CUDA also the matching cudart-llama-bin-win-cuda-13.5-x64.zip, which carries the CUDA runtime libraries. Unzip both into one folder and run:

.\llama-server.exe -m models\my-model.gguf -c 8192

There are also Windows on ARM builds, including one for Adreno GPUs.

Docker

The server images are server for CPU and server-cuda, server-cuda13, server-vulkan, server-rocm and server-intel for GPUs. The documented command still maps port 8080, because the images keep that port inside the container:

docker run -p 8080:8080 -v /path/to/models:/models --gpus all \
  ghcr.io/ggml-org/llama.cpp:server-cuda \
  -m /models/my-model.gguf --host 0.0.0.0 --port 8080 --n-gpu-layers 99

Start a model with -hf or -m

To download a model from Hugging Face and serve it:

llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M

The tag after the colon chooses the quantisation. It is optional and defaults to Q4_K_M, or the first file in the repository if no Q4_K_M file exists. If the repository includes a multimodal projector, it is downloaded too; add --no-mmproj to skip it. A gated repository needs -hft or the HF_TOKEN environment variable.

To serve a file you already have:

llama-server -m ./models/my-model.gguf -c 16384 --alias my-model

Then test it from another terminal:

curl http://localhost:9931/health
curl http://localhost:9931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "my-model", "messages": [{"role": "user", "content": "Hello"}]}'

/health returns 503 while the model loads and {"status": "ok"} once it is ready. Two flags help on restricted networks: --offline forces the cache and prevents network access, and HTTPS_PROXY is honoured for model downloads behind a corporate proxy.

Not sure a model will fit? Our VRAM calculator estimates memory for a model, quantisation and context length, and the quantisation guide explains what Q4_K_M trades away.

llama.cpp server flags that matter

The README’s flag list runs to well over 200 rows. These are the ones that decide whether a set-up works.

FlagDefaultWhat it does
-m, --modelnonePath to a GGUF file
-hf, --hf-repononeHugging Face repository, with optional :quant
-c, --ctx-size0Context size; 0 means read it from the model
-fit, --fitonAdjusts unset arguments to fit device memory; --fit-ctx sets the smallest context it may pick (4096)
-ngl, --gpu-layersautoLayers to keep in VRAM: a number, auto or all
-np, --parallel-1 (auto)Number of server slots, meaning requests handled at once
--host127.0.0.1Listen address; accepts several addresses or a UNIX socket path
--port9931Listen port
--api-key, --api-key-filenoneRequire a key; several keys allowed
-a, --aliasnoneModel name or names reported to API clients
--jinjaenabledUse the Jinja chat template engine, which tool calling needs
-fa, --flash-attnautoFlash Attention on, off or auto
-to, --timeout3600Read and write timeout in seconds
--metricsdisabledPrometheus-compatible /metrics endpoint
--slotsenabledSlot monitoring endpoint
--sleep-idle-seconds-1 (disabled)Unload the model after this much idle time
--embedding, --rerankdisabledServe an embedding model, or enable the reranking endpoint

Three behaviours catch people out.

Context and --fit. With -c unset, the server reads the context length from the model, and --fit, which is on by default, adjusts unset arguments so the load fits in device memory. The README’s router section says an instance then takes the largest context that fits at load time. That is convenient on one machine and unpredictable across machines, so set -c explicitly for anything you share or script.

Context and slots. When you set -np yourself, the context is shared out between slots. Hermes Agent’s documentation gives the standard example: -c 64000 -np 4 leaves 16K tokens per slot. With slots left on auto, the server uses a single unified KV buffer shared across sequences, and --kv-unified-per-slot caps what one slot may use. For a coding agent that needs 64K, either run one slot or multiply the context.

Environment variables. Most flags have an environment form, such as LLAMA_ARG_CTX_SIZE, LLAMA_ARG_N_GPU_LAYERS, LLAMA_ARG_PORT and LLAMA_API_KEY. A command-line argument wins when both are set.

The default port changed from 8080 to 9931

For years llama-server listened on port 8080. The project added a notice about an upcoming change on 3 August 2026 (pull request 26508) and switched the default on 9 October 2026 (pull request 30159, first shipped in build b11521). The README now says the server listens on 127.0.0.1:9931 by default.

What this breaks, and how to fix it:

  • Client configs and tutorials that assume 8080. Many coding agents document a llama.cpp provider on port 8080. Pass --port 8080 to the server, or change the client’s base URL to http://127.0.0.1:9931/v1.
  • Docker. Nothing changes: the pull request kept 8080 inside the container so tools built on the base image keep working, and the documented commands pass --port 8080 explicitly.
  • Auto-detection. Tools that probe 8080 for a running server will not find one on the new default. Hermes Desktop is one; our Hermes Desktop set-up guide shows the config line that points it at 9931.

The safest habit is to write the port into every start command and unit file. Our set-up guides for OpenCode, Claude Code and the Pi coding agent each show the client side for their tool.

llama.cpp server API: OpenAI and Anthropic endpoints

EndpointPurpose
GET /healthReadiness check; public even when an API key is set
GET /v1/modelsThe loaded model, with input and output modalities
POST /v1/chat/completionsOpenAI-compatible chat, streaming, tools, JSON schema output
POST /v1/completionsOpenAI-compatible text completion
POST /v1/responsesOpenAI-compatible Responses API
POST /v1/embeddingsOpenAI-compatible embeddings
POST /v1/messagesAnthropic-compatible Messages API, with /v1/messages/count_tokens
POST /rerankingRerank documents against a query (needs --rerank)
POST /completion, /tokenize, /detokenize, /infillNative llama.cpp endpoints
GET /props, /slots, /metricsServer properties, slot state, Prometheus metrics

The README is careful about compatibility: it makes no strong claim to match the OpenAI or Anthropic specifications, only that the routes work with many applications. In practice, point an OpenAI client at the /v1 base URL:

import openai

client = openai.OpenAI(
    base_url="http://localhost:9931/v1",
    api_key="sk-no-key-required",
)

For Anthropic-style clients, the Messages route accepts system, tools, tool_choice and streaming, and the README’s example sends the key in an x-api-key header. Fields llama.cpp has but the vendor APIs lack, such as seed or min_p, go in an extra_params object on the Messages and Responses routes.

Schema-constrained output works through response_format on chat completions, with either json_object or a JSON schema. Recent builds also list a WebSocket /v1/realtime route for transcription and conversation and an OpenAI-compatible speech route.

The built-in web UI

Open http://localhost:9931 in a browser and you get a chat interface served by the same process. It is enabled by default; --no-ui turns it off for API-only deployments. Defaults for the UI can be set with --ui-config or --ui-config-file, and the older --webui- spellings still work as aliases.

The UI is also where the server’s agent features surface. With --tools, the model can call built-in tools from the UI: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file and get_info. The --agent flag enables all of them plus a CORS proxy. The README marks these as experimental and says not to enable them in untrusted environments, because they give the model read and write access to the host’s files and a shell. --tools-runtime can run them in a Docker or Podman container or over SSH instead of on the host.

Router mode: several models on one server

Start llama-server with no model and it runs as a router: a parent process that starts a child server per model and forwards requests.

llama-server --models-dir ./models --models-max 2

Models can come from three places: the download cache, a directory given with --models-dir, or an INI file given with --models-preset. A preset file sets per-model arguments:

version = 1

[*]
c = 8192

[coder]
model = /srv/models/coder-Q4_K_M.gguf
n-gpu-layers = all
c = 65536

Requests are routed by the model field in the JSON body, or a model query parameter on GET endpoints. Models load on first use unless you pass --no-models-autoload. --models-max limits how many stay loaded (default 4), unloading the least recently used, and --models-memory-margin makes the router measure each model and free memory before loading another. GET /models lists models and their state, and POST /models/load and /models/unload control them.

Router mode is useful on a workstation that switches between a chat model, a coding model and an embedding model. Be aware of its status, though: the project’s security policy lists the router among features not recommended for untrusted environments, and an issue opened on 11 October 2026 asks for authentication between the router and its instances.

Tool calling and Jinja templates

Tool calling depends on the model’s chat template. The server’s Jinja template engine is enabled by default in current builds (--jinja, with --no-jinja to switch it off), so the old advice to add --jinja is now a precaution, not a requirement. The Anthropic route’s documentation still words it as needing --jinja.

The function-calling doc describes two paths:

  • Native handlers for formats the server recognises, including Llama 3.x, Functionary, Hermes 2 and 3, Qwen 2.5 and Qwen 2.5 Coder, Mistral Nemo, Firefunction v2, Command R7B and GPT-OSS.
  • A generic handler for everything else. It works, but the doc says it may use more tokens and be less efficient. The log line Chat format: Generic tells you it is in use.

If a model’s embedded template is wrong or missing tool support, override it with --chat-template-file. To check what the server loaded, look at the chat_template property at /props. Parallel tool calls are off by default and enabled per request with "parallel_tool_calls": true. For reasoning models, --reasoning-format controls whether thinking is returned in reasoning_content, and --reasoning-budget caps it.

Template support matters as much as model size here. Our guide to local models for tool calling lists models that handle agent loops reliably.

External tool servers in llama-server

llama-server can act as an MCP host. Pull request 26062, merged on 25 July 2026, added support for MCP servers over the stdio transport. Declare them in a Cursor-compatible JSON file:

{
  "mcpServers": {
    "files": { "command": "/usr/local/bin/my-mcp-server", "args": ["--root", "/srv/docs"] }
  }
}
llama-server -m model.gguf --mcp-servers-config mcp.json

Each entry accepts command, args, env, cwd and timeout_ms (30 seconds by default). The server starts each MCP server once at launch to list its tools, stops it, and starts it again when a tool is called. Tools appear to the model and the web UI as <server>_<tool>.

The limits are stated plainly in the README:

  • Stdio only. Remote HTTP MCP servers are not supported by this feature. The separate --ui-mcp-proxy flag only lets the web UI reach public MCP servers that lack CORS headers, and it refuses local and private addresses.
  • Same privileges. An MCP server runs as a child process with llama-server’s own permissions, so declare only commands you trust.
  • Experimental. The flag is labelled experimental, and enabling it narrows --cors-origins to localhost by default. Tools and MCP on a non-loopback address need an API key unless you pass --allow-unauthenticated-tools.

Note the direction: this makes llama-server a client of MCP servers. It does not turn llama-server into an MCP server that other agents connect to. The web UI can also connect to MCP servers by URL from the browser; our guide to MCP with local model runtimes covers that side and compares it with LM Studio and Open WebUI.

Run llama-server as a service

llama.cpp does not ship a service installer, so use the host’s own tools. On Linux, a minimal systemd unit looks like this; it is our example, not an upstream file:

[Unit]
Description=llama-server
After=network-online.target

[Service]
User=llama
ExecStart=/usr/local/bin/llama-server -m /srv/models/model.gguf -c 32768 --host 127.0.0.1 --port 9931 --api-key-file /etc/llama/keys --metrics
Restart=on-failure

[Install]
WantedBy=multi-user.target

With Docker Compose, the README’s pattern is to configure the server through environment variables:

services:
  llamacpp-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    ports:
      - 8080:8080
    volumes:
      - ./models:/models
    environment:
      LLAMA_ARG_MODEL: /models/my_model.gguf
      LLAMA_ARG_CTX_SIZE: 4096
      LLAMA_ARG_N_PARALLEL: 2
      LLAMA_ARG_ENDPOINT_METRICS: 1
      LLAMA_ARG_PORT: 8080

For a server that sits idle most of the day, --sleep-idle-seconds unloads the model and its cache after the given idle time and reloads on the next request. Health, props, models and metrics calls do not wake it. --metrics exposes token throughput, busy slots and deferred requests for Prometheus, and --log-file writes logs to a file.

Security when exposing llama-server

The defaults are safe for one machine and nothing more: the server binds to 127.0.0.1, has no API key, and reflects any Origin header for CORS. Before changing --host, work through this list.

  1. Set an API key. Use --api-key-file so keys stay out of the process list. Remember that /health stays public.
  2. Add TLS. A build with -DLLAMA_OPENSSL=ON accepts --ssl-key-file and --ssl-cert-file. Most teams terminate TLS at a reverse proxy instead.
  3. Put a reverse proxy in front. The README’s CORS table recommends an API key plus a reverse proxy for public deployments, and a specific --cors-origins value on a local network.
  4. Keep agent features off. Do not combine --tools, --agent or MCP with a network-reachable address unless every client is trusted.
  5. Leave --connect alone. Added on 19 October 2026 (pull request 28277), it opens an experimental peer-to-peer tunnel to the server from outside the LAN.
  6. Treat models as untrusted input. The security policy advises running models from unknown sources in a container or VM and checking the hash of every downloaded file.
  7. Know what the project covers. The policy excludes the web UI, experimental features, the router, MCP and most denial-of-service bugs from its vulnerability scope, and says the private disclosure programme is currently disabled.

An API key shared by everyone is not user management. There are no accounts, quotas per person or audit records in llama-server, and its own policy puts tenant isolation on the operator.

Where the llama.cpp server docs live

There is no separate documentation site for the server. The reference is in the repository:

  • tools/server/README.md: every flag, generated from the binary’s help text, and every endpoint.
  • docs/function-calling.md: tool calling, templates and examples.
  • docs/install.md, docs/docker.md and docs/build.md: packages, images and compiling.
  • SECURITY.md: the security policy.
  • Issue 9291: the server’s REST API changelog.

Because the flag list is generated from the code, llama-server --help on your own build is the most accurate reference for that build.

How VDF AI fits

llama-server is a model server: it turns a GGUF file into an API on one machine. A team that wants many people and agents to use private models needs things it leaves out by design, such as sign-in, access rules per group, approved-model lists and a record of who asked what. VDF AI does not ship, host, resell or support llama.cpp.

VDF AI is an on-premises and private-cloud platform for governed agents, multi-agent Networks and private RAG, with role-based access control and audit logs on every plan. VDF AI Router registers Ollama and custom on-premises deployments next to the cloud models your policy approves, and its air-gap mode keeps routing on local models only. VDF AI documents no llama-server integration, so treat that pairing as something to test in a pilot.

Sources

Repository files, pull requests and release pages read on 14 November 2026 (latest build that day: b12387).


Moving from one model server to private AI for a whole team? See how VDF AI Router applies model policy, or book a demo.

Frequently asked questions

What is llama-server?

llama-server is the HTTP server included with llama.cpp, the MIT-licensed C and C++ inference engine for GGUF models. One binary loads a model and exposes OpenAI-compatible chat, completions, responses and embeddings routes, an Anthropic-compatible Messages route, a reranking endpoint, monitoring endpoints and a browser chat UI. It handles several requests at once through parallel slots and continuous batching, runs on CPU and on GPUs through CUDA, Metal, Vulkan and other back ends, and can load several models in router mode.

What port does llama-server use?

Port 9931 on 127.0.0.1 by default. The default was 8080 until build b11521, published on 9 October 2026, when pull request 30159 changed it; a notice about the change had been in the server since 3 August 2026. The official Docker images still use 8080 inside the container. If a client or older tutorial expects 8080, start the server with --port 8080 or set LLAMA_ARG_PORT, and state the port explicitly in scripts either way.

How do I start llama-server with a model from Hugging Face?

Pass a repository name to -hf, for example llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M. The part after the colon selects the quantisation and defaults to Q4_K_M, falling back to the first file in the repository. The server downloads the file to its cache, fetches a multimodal projector if the repository has one, and starts listening. For a file already on disk, use -m with the path. Add --offline to force the cache and block network access.

Does llama-server have an API key?

Only if you set one. By default the server has no authentication and listens on 127.0.0.1, so only programs on the same machine can reach it. The --api-key flag takes one key or a comma-separated list, --api-key-file reads keys from a file, and the LLAMA_API_KEY environment variable does the same. Clients then send the key as a bearer token. The health endpoint stays public. For anything beyond one machine, add TLS and a reverse proxy as well.

Can llama-server start external tool servers?

Yes, as an experimental feature. Since pull request 26062, merged on 25 July 2026, llama-server can start MCP servers over the stdio transport and expose their tools to the model and the web UI. You declare them in a Cursor-compatible JSON file and pass it with --mcp-servers-config. Only stdio is supported, each MCP server runs with the same privileges as llama-server, and the README says not to enable it in untrusted environments.

Can llama-server serve more than one model?

Yes, in router mode. Start llama-server without a model and it becomes a router that loads models on demand and forwards each request by the model field in the JSON body. Models come from the cache, a folder given with --models-dir, or an INI preset file given with --models-preset. By default up to four models stay loaded, and the least recently used one is unloaded first. The security policy lists the router among features not recommended for untrusted environments.

Filed under
llama.cppLLM inferencelocal LLMopen-source AIlocal AI infrastructureAI security
On-Prem AI

Plan your on-prem AI deployment

Book an architecture call and we will scope a private, on-prem AI deployment for your environment — integrations, hardware, and governance included.

Keep reading