How to Run OpenClaw and Hermes Agent on a Local LLM

An OpenClaw local LLM setup points the OpenClaw gateway, or Hermes Agent, at a model server on your own hardware instead of a hosted API. Prompts and files stay on your machine and per-token charges disappear, but the model must produce reliable tool calls and hold a 32K to 64K token context in the memory you have.

Both harnesses support local models, and both document the setup in detail. The details differ in ways that matter: the two projects talk to Ollama through different endpoints, and each has its own view of how much context an agent needs. This guide follows the official documentation of each project and model, checked on 27 September 2026. If local models are new to you, start with what a local LLM is.

Why run an agent harness on a local model

A harness sends far more to the model than a chat window does: system prompts, skill text, tool schemas, file contents, command output and memory. On a hosted API every one of those tokens leaves your network and is billed. A local model keeps them on hardware you control and turns the running cost into electricity and depreciation.

The trade-offs are real. Smaller open-weight models call tools less reliably than frontier APIs, long contexts consume GPU memory quickly, and you now operate a model server. Privacy also depends on configuration. Ollama models with a :cloud suffix run on ollama.com, and launching OpenClaw through ollama launch turns on Ollama web search automatically. Decide which of those you want before calling the setup private. If the agent will touch work systems, read our OpenClaw security review first.

Choose a serving stack: Ollama, vLLM or llama.cpp

Two Ollama defaults catch people out. Its context length depends on VRAM: 4K below 24 GiB, 32K from 24 to 48 GiB, and 256K from 48 GiB, while its context guide recommends at least 64,000 tokens for agents and coding tools (OLLAMA_CONTEXT_LENGTH=64000 ollama serve). And its KV cache can be quantized to q8_0, roughly halving cache memory, through OLLAMA_KV_CACHE_TYPE when flash attention is on.

For vLLM, take the parser name from the model card: the Qwen3.5 and Qwen3.6 cards use qwen3_coder, Devstral Small 2 uses mistral, GLM-4.7-Flash uses glm47, and vLLM’s tool-calling docs list openai for gpt-oss. The trade-offs between the servers are covered in our local LLM deployment handbook.

Point OpenClaw at the local server

OpenClaw’s Ollama provider uses Ollama’s native API, and the docs warn against the /v1 URL because it breaks tool calling. The quickest path is ollama launch openclaw, which installs OpenClaw if needed and configures the provider. To configure it by hand:

{
  models: {
    providers: {
      ollama: {
        baseUrl: "http://127.0.0.1:11434", // no /v1
        apiKey: "ollama-local",
        api: "ollama",
        timeoutSeconds: 300,
        models: [{ id: "qwen3.5:9b", name: "qwen3.5:9b", contextTokens: 32768, params: { num_ctx: 32768 } }],
      },
    },
  },
  agents: { defaults: { model: { primary: "ollama/qwen3.5:9b" } } },
}

Keep contextTokens (OpenClaw’s input budget) and params.num_ctx (Ollama’s request context) aligned. Local discovery normally caps context at 32,768 tokens, while Ollama’s OpenClaw page recommends at least 64K for local models, so raise both together once the hardware allows.

For vLLM, OpenClaw uses the vllm provider with api: "openai-completions" and a base URL such as http://127.0.0.1:8000/v1; setting VLLM_API_KEY lets it discover models from /v1/models. For LM Studio, run lms server start --port 1234, then openclaw onboard and choose LM Studio. Verify with:

openclaw models list --provider ollama
openclaw infer model run --model ollama/qwen3.5:9b --prompt "Reply with exactly: ok"

Point Hermes Agent at the local server

Hermes Agent treats any server that implements /v1/chat/completions as a custom endpoint, so for Ollama it uses the OpenAI-compatible path. Run hermes model, choose “Custom endpoint”, or edit ~/.hermes/config.yaml as the provider docs show:

model:
  default: "qwen3.6"
  provider: "custom"
  base_url: "http://localhost:11434/v1"

Hermes states that agentic work with tools needs at least 64,000 tokens of context, so raise Ollama’s default through OLLAMA_CONTEXT_LENGTH or a Modelfile with PARAMETER num_ctx 64000. The same rule shapes other servers. With llama-server, pass --jinja and -c 64000, and note that parallel slots split the context: -c 64000 -np 4 leaves each slot 16K, below Hermes’ minimum. With vLLM, use http://localhost:8000/v1 and start the server with the tool-calling flags.

On the desktop app, Settings, Providers, Local Models installs a managed llama.cpp runtime, rates each catalog model for fit on your GPU, and never offers builds below 4-bit. ollama launch hermes is the equivalent one-command path for Ollama users.

Isolate the agent with Docker

Both harnesses can run commands on the host by default, so put a container boundary around execution before you connect real accounts.

• OpenClaw offers two layers: a containerized gateway (./scripts/docker/setup.sh, images from ghcr.io/openclaw/openclaw, with an --offline mode for air-gapped hosts) and a separate tool sandbox under agents.defaults.sandbox with Docker or Podman backends. The sandbox is off until you set mode to non-main or all.
• Hermes Agent can run tool commands in a hardened container with terminal.backend: docker, or run entirely inside the nousresearch/hermes-agent image with its data mounted at /opt/data.

Containers change networking. Ollama binds 127.0.0.1 by default, so a containerized gateway cannot reach it until you change OLLAMA_HOST, and OpenClaw’s Ollama recipes suggest checking exactly this when curl works but OpenClaw does not. Expose the model server on a private interface only, and filter published container ports in the DOCKER-USER chain as the OpenClaw docs describe.

Add MCP servers without flooding the context

Both harnesses are MCP clients. In OpenClaw, openclaw mcp add <name> --command ... or --url ... writes an entry under mcp.servers, openclaw mcp doctor <name> --probe confirms it answers, and --include limits which tools are exposed. In Hermes Agent, servers go under mcp_servers in config.yaml with command and args, and per-server include and exclude filters trim the tool list.

Filtering matters more with local models, because every tool schema takes context. OpenClaw applies structured Tool Search automatically for local Ollama models, loading schemas only when needed. Either way, expose only the tools a task requires.

Pick a model that calls tools reliably

Choose on three criteria before any benchmark: the model card documents native tool calling your server can parse, the native context comfortably exceeds 64K, and the quantized weights leave room for the KV cache on your GPU. The families below meet the first two, per their model cards (checked 27 September 2026):

The Qwen cards advise keeping at least 128K of context to preserve thinking behaviour, which costs memory. No published score substitutes for a trial on your own tasks, so log failed and malformed tool calls over a fixed set of real requests. For the procurement side, see our guide to sizing GPU memory.

Hardware fit by VRAM tier

Weight memory follows from simple arithmetic: parameters (billions) × bits per weight ÷ 8 = gigabytes. llama.cpp’s quantization table puts Q4_K_M at 4.89 bits per weight and Q8_0 at 8.5; BF16 is 16. Mixture-of-experts models must hold every expert in memory, so size them by total parameters.

Then add the KV cache. Per token it takes 2 × layers × KV heads × head dimension × bytes per value. Devstral Small 2’s config lists 40 layers, 8 KV heads and a head dimension of 128, so at 16-bit that is 2 × 40 × 8 × 128 × 2 = 163,840 bytes per token, or about 10.7 GB at 64K tokens. Add the 14.7 GB of weights and the total passes a 24 GB card before runtime overhead. An 8-bit KV cache (about 5.4 GB) or a 32K window brings it to roughly 20 GB. The local LLM deployment handbook extends this method to concurrency and multi-GPU serving.

How VDF AI fits

A harness on a workstation serves one operator. When local models need to serve an organisation, VDF AI provides a governed alternative deployed on-premises, in a private cloud or air-gapped. It does not run or connect to OpenClaw or Hermes Agent; for how the approaches differ, see VDF AI compared with OpenClaw, the Hermes and OpenClaw comparison and our explainer on agent harnesses.

The VDF AI Router puts one REST endpoint in front of local open-weight models, registers Ollama and custom on-premises deployments alongside any external endpoints policy allows, and in air-gap mode disables external APIs so routing stays on local models. Workspace admins in VDF AI Networks set which models each workflow may use.