AI Infrastructure

Qwen Code Local LLM Setup: Ollama, LM Studio, llama.cpp and vLLM, and What Gemini CLI Can Do (2026)

How to run Qwen Code, Alibaba's open-source terminal coding agent, on a model served from your own hardware: settings.json entries for Ollama, LM Studio, llama.cpp and vLLM, the context and tool-calling settings that decide whether it works, which Qwen coder models fit which machine, and what Google's Gemini CLI does and does not support for local models. Verified against Qwen Code 0.27.3 and Gemini CLI 0.68.0 documentation on 14 November 2026.

Qwen Code runs on a local LLM through its OpenAI-compatible provider: add the model to modelProviders in settings.json with the /v1 address of Ollama, LM Studio, llama.cpp or vLLM, select it with /model, and set contextWindowSize to what the server loaded. Gemini CLI has no local provider; it reaches local models only through a proxy that speaks the Gemini API.

Qwen Code is Alibaba’s open-source terminal coding agent, published under Apache 2.0 in the QwenLM/qwen-code repository. Its README says it was originally based on Google’s Gemini CLI 0.8.2 and stopped syncing with upstream from version 0.1. That history explains the search pattern: people who liked Gemini CLI’s interface but wanted a local model ended up here, because Qwen Code speaks the OpenAI, Anthropic and Gemini protocols and Gemini CLI speaks only Google’s.

The configuration keys and defaults below come from the Qwen Code and Gemini CLI documentation in their repositories, read on 14 November 2026 against Qwen Code 0.27.3, released the day before, and Gemini CLI 0.68.0 of 10 November. A Qwen Code 1.0 preview exists on a separate branch with its own docs; this guide follows the main version. Which model to run is covered in our review of open-weight coding models, and the same job for OpenAI’s agent is in our Codex guide for local models.

Qwen Code and Gemini CLI: what works with local models

Qwen CodeGemini CLI
Native local model supportYes, through the openai provider and a local baseUrlNo
ProtocolsOpenAI Chat Completions and Responses, Anthropic, GeminiGemini API and Vertex AI
Local servers in the docsOllama, vLLM, LM Studio, plus SGLang for Qwen3.xNone; LiteRT-LM for a routing classifier only
Custom base URLbaseUrl on each model entry, or OPENAI_BASE_URLGOOGLE_GEMINI_BASE_URL, for Gemini-format endpoints
Account needed for local useNoA Gemini API key value, which a proxy can supply
Who it is for todayAnyoneGemini Code Assist Standard and Enterprise, per its README

The last row is new. Gemini CLI’s README now opens with a notice that free individual access and Google AI Pro and Ultra plans moved to Antigravity CLI on 18 June 2026, and that Gemini CLI is maintained for enterprise Code Assist workflows on Google Cloud and for paid API and Vertex keys.

How Qwen Code connects to a local model

Install it with the project’s script, with npm (Node.js 22 or newer) or with Homebrew:

npm install -g @qwen-code/qwen-code@latest

Qwen Code reads ~/.qwen/settings.json. Four parts of that file matter for a local model:

  • modelProviders maps a provider id to an array of model entries. For a local server the id is openai. Each entry needs an id, and normally a baseUrl, a name and an envKey.
  • envKey is the name of an environment variable, not a key. Local servers need no real key, so the docs use placeholders such as OLLAMA_API_KEY="ollama". The value can live in the env block of the same file, in ~/.qwen/.env or in your shell.
  • security.auth.selectedType set to openai tells Qwen Code which protocol to start with, so it does not open the /auth menu.
  • model.name sets the default; /model switches during a session.

Edits to modelProviders are picked up by a running session without a restart. The interactive route is /auth, then Custom Provider, which the docs describe as the option for a local server or proxy.

For a first test you can skip the file. The docs list three variables for the OpenAI-compatible protocol:

export OPENAI_API_KEY="ollama"
export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_MODEL="qwen3-coder:30b"
qwen

The file is still worth writing, because it is the only place to pin the context window and timeouts per model.

Qwen Code with Ollama

  1. Pull a model with tool support:

    ollama pull qwen3-coder:30b
  2. Raise the context on the server. Ollama’s OpenAI-compatible API cannot set a context size per request, and its default is 4K tokens on machines with less than 24 GiB of VRAM. Ollama recommends at least 64,000 tokens for agents and coding tools:

    OLLAMA_CONTEXT_LENGTH=65536 ollama serve
  3. Add the model to ~/.qwen/settings.json. This follows the local example in Qwen Code’s docs, with a current model:

    {
      "env": {
        "OLLAMA_API_KEY": "ollama"
      },
      "modelProviders": {
        "openai": [
          {
            "id": "qwen3-coder:30b",
            "name": "Qwen3-Coder 30B (Ollama)",
            "envKey": "OLLAMA_API_KEY",
            "baseUrl": "http://localhost:11434/v1",
            "generationConfig": {
              "timeout": 300000,
              "streamIdleTimeoutMs": 600000,
              "contextWindowSize": 65536
            }
          }
        ]
      },
      "security": {
        "auth": {
          "selectedType": "openai"
        }
      },
      "model": {
        "name": "qwen3-coder:30b"
      }
    }
  4. Start qwen in a project folder and check the server while it works:

    ollama ps

    The CONTEXT column should show 65536, the same number as contextWindowSize.

Ollama’s launch command covers several coding agents, but Qwen Code is not on its integrations list, so there is no one-line setup.

Qwen Code with LM Studio

  1. Load the model with the context you want, then start the server:

    lms load qwen/qwen3-coder-30b --context-length 65536
    lms server start
  2. Add an entry to the openai array, using the identifier LM Studio serves as the id:

    {
      "id": "qwen/qwen3-coder-30b",
      "name": "Qwen3-Coder 30B (LM Studio)",
      "envKey": "LMSTUDIO_API_KEY",
      "baseUrl": "http://localhost:1234/v1",
      "generationConfig": {
        "timeout": 300000,
        "contextWindowSize": 65536
      }
    }
  3. Add "LMSTUDIO_API_KEY": "lm-studio" to the env block, as the docs’ example does, and select the model with /model.

If LM Studio loads a model on demand, it uses the context length saved for that model in LM Studio, which may be smaller than the value in your entry. Load the model explicitly, or change its saved default in LM Studio.

Qwen Code with llama.cpp and vLLM

llama.cpp

Current builds of llama-server listen on port 9931 unless told otherwise and enable the Jinja chat template, which OpenAI-style tool calling depends on. Pass the port, an alias and a context size; the rest of the flags are in our guide to running the llama.cpp server:

llama-server -hf ggml-org/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M \
  --alias qwen3-coder-30b --host 127.0.0.1 --port 8080 -c 65536

The entry is the same shape as before, with "id": "qwen3-coder-30b", "baseUrl": "http://127.0.0.1:8080/v1" and "contextWindowSize": 65536. Set a placeholder variable for envKey, such as the docs’ VLLM_API_KEY="not-needed".

vLLM

For a GPU server shared by a team, the Qwen3-Coder-Next model card gives this vLLM command, to which we add a served name:

vllm serve Qwen/Qwen3-Coder-Next --port 8000 --tensor-parallel-size 2 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --served-model-name qwen3-coder-next
{
  "id": "qwen3-coder-next",
  "name": "Qwen3-Coder-Next (vLLM)",
  "envKey": "VLLM_API_KEY",
  "baseUrl": "http://localhost:8000/v1",
  "generationConfig": {
    "contextWindowSize": 262144
  }
}

Qwen Code’s docs say to set contextWindowSize to the server’s maximum model length, which vLLM reports as max_model_len on its /v1/models route. The card notes that the default is 256K and suggests a smaller value such as 32,768 if the server fails to start. If the server uses a self-signed certificate behind https://, Qwen Code’s troubleshooting page covers the Node.js error that results. For the differences between these runtimes, see our vLLM, Ollama and llama.cpp comparison.

Context settings and tool-calling caveats

Why the context window must be pinned

Qwen Code infers the context window from the model id. A name it does not recognise gets a generic 200K window, and a recognised name gets whatever the built-in lookup holds for that model. Neither is likely to match a local server that loaded 32K or 64K. Auto-compaction triggers at a fraction of the window, 0.85 by default under context.autoCompactThreshold. If Qwen Code believes it has 200K tokens and the server holds 65,536, compaction never starts and the server runs out first. Setting contextWindowSize on the entry fixes this.

Timeouts

Local models are slow to start and sometimes slow between chunks. The docs give three controls:

  • timeout in generationConfig, the request timeout in milliseconds.
  • streamIdleTimeoutMs, how long the model may stay silent between streamed chunks. It overrides the global QWEN_STREAM_IDLE_TIMEOUT_MS, and 0 disables the guard.
  • A separate 15-minute cap on any one stream, raised or disabled with QWEN_STREAM_MAX_LIFETIME_MS.

Tool calling

Qwen Code sends tool definitions on every request, so the server must parse the model’s tool calls into the OpenAI format.

  • Ollama and LM Studio handle this for models that carry the tools capability. qwen3-coder is tagged for tools in Ollama’s library.
  • llama.cpp needs the Jinja template, on by default. The server README adds that some models need a --chat-template-file override to get a template compatible with tool use.
  • vLLM needs --enable-auto-tool-choice and a --tool-call-parser that matches the model. The Qwen3-Coder-Next card uses qwen3_coder; Qwen Code’s docs give qwen3_xml with --reasoning-parser qwen3 as an example for Qwen3.x models and warn that parser names change between server releases.
  • Renamed Qwen3.x models lose thinking control. If a vLLM or SGLang server exposes a Qwen3.x thinking model under a name that does not begin with qwen, the docs say to declare "capabilities": { "reasoning": { "profile": "qwen-chat-template" } } on the entry. Otherwise side requests, such as the permission classifier used by Auto mode, keep thinking switched on.

The Qwen3-Coder models avoid the last problem: their cards say they support only non-thinking mode. General guidance on models that call tools reliably is in our tool-calling model guide.

Testing the setup and common errors

  1. List the model ids the server exposes. Ollama, LM Studio, llama-server and vLLM all answer the OpenAI-style models route:

    curl http://localhost:11434/v1/models
  2. Run one prompt in headless mode with the model named explicitly:

    qwen -m qwen3-coder:30b -p "List the top-level folders in this repository"
  3. Open an interactive session, run /model and confirm the entry is listed under the name you gave it.

SymptomLikely causeFix
The model is missing from /modelProvider id is not openai or another supported id, so the whole entry was skipped with a warningUse openai, or map a custom id with providerProtocol and restart
Two entries, one shownSame protocol, id and baseUrl; the first winsChange the id or the URL
Connection refusedWrong port, or baseUrl without /v1Check the server; llama-server defaults to 9931
The agent loses the task after a few stepsServer context smaller than contextWindowSizeRaise the server context or lower the setting
Tool calls appear as textServer is not parsing tool callsEnable the tool-call parser or the Jinja template
Streams stop after a silenceIdle guard on a slow modelRaise streamIdleTimeoutMs

The id must be exactly what the server reports. Ollama uses tags such as qwen3-coder:30b, LM Studio uses paths such as qwen/qwen3-coder-30b, and llama-server reports the file path unless you pass --alias.

Which local Qwen coder models fit which hardware

ModelOllama tag and downloadParametersMemory with contextFits
Qwen3-Coder-30B-A3B-Instructqwen3-coder:30b, 19 GB30.5B, 3.3B active≈ 23 GB at 4-bit with one 64K session (our arithmetic)24 GB GPU with a shorter window; 32 GB GPU or Mac
Qwen3.8-27Bqwen3.8:27b, 18 GB27B, dense≈ 23 GB with 64K (Cline’s estimate)24 GB GPU at the limit; 32 GB comfortable
Qwen3.6 35B A3Bqwen3.6:35b-a3b, 23 GB35B mixture-of-experts build, per its name≈ 26 GB with 64K (Cline’s estimate)32 GB GPU or Mac
Qwen3-Coder-Nextqwen3-coder-next, 52 GB80B, 3B active≈ 56 GB with 64K (Cline’s estimate)64 GB unified memory or more; one 80 to 96 GB GPU

Download sizes are from Ollama’s library. The estimates marked Cline come from the recommended-model table in Cline Desktop’s documentation, which our Cline set-up guide also tabulates, and the first row from the memory arithmetic in our coding-model review. Both Qwen3-Coder cards list a native context of 262,144 tokens under Apache 2.0, and both say to reduce the context to a value such as 32,768 if you run out of memory.

The two mixture-of-experts coder models activate about 3B parameters per token, which keeps them quick for their size. Qwen3.8-27B is a general model with thinking on by default; its builds and memory tiers are in our Qwen3.8 hardware guide. To test a combination before downloading, use the VRAM calculator.

One sampling note: the Qwen3-Coder-30B card suggests a temperature of 0.7 with top_p 0.8, and the Qwen3-Coder-Next card suggests 1.0 with top_p 0.95. Qwen Code passes such values through samplingParams in generationConfig if you want to set them.

Gemini CLI local LLM: what is and is not possible

Gemini CLI’s authentication page offers three methods: sign in with Google, a Gemini API key, or Vertex AI. There is no Ollama provider, no OpenAI-compatible provider and no setting that names a local model as the main model.

What the documentation does contain:

  • A local routing classifier. The experimental gemini gemma setup command downloads a Gemma 3 1B model of about 1 GB and serves it with LiteRT-LM. It classifies each request as simple or complex and sends it to Gemini Flash or Gemini Pro. The answer still comes from a hosted model.
  • A base URL override. GOOGLE_GEMINI_BASE_URL overrides the base URL for Gemini API requests when you use API-key authentication. It must use HTTPS unless it points at localhost, 127.0.0.1 or [::1]. GOOGLE_VERTEX_BASE_URL does the same for Vertex AI.

The override is a redirect, not a protocol switch. Whatever answers at that address has to speak the Gemini API. Ollama, LM Studio, llama.cpp and vLLM do not, so pointing the variable straight at them fails.

Gemini CLI with Ollama through LiteLLM

LiteLLM’s proxy accepts Gemini-format requests and publishes a Gemini CLI tutorial. It sets two variables:

export GOOGLE_GEMINI_BASE_URL="http://localhost:4000"
export GEMINI_API_KEY=sk-<your-litellm-api-key>

To serve a non-Gemini model, the tutorial uses model_group_alias, which maps the Gemini model name the CLI requests to another entry. Its examples route to hosted providers. Combining it with LiteLLM’s documented Ollama provider gives a local route:

model_list:
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3-coder:30b
      api_base: http://localhost:11434
router_settings:
  model_group_alias: {"gemini-3.1-pro-preview": "local-qwen"}

The alias key has to match the model name your Gemini CLI version sends; setting GEMINI_MODEL makes that predictable. Three cautions apply. Google does not document or support this arrangement, and we assembled the local variant from two LiteLLM pages without a statement from either project that the pair is tested. Gemini CLI’s README says its harness is not tuned even for newer Gemini models, so prompts written for Gemini will not suit a 30B local model well. And the proxy becomes one more component to secure; our LiteLLM alternatives review compares the gateways that can fill that role.

What about Antigravity CLI?

Antigravity CLI is where Google sent individual Gemini CLI users. Its model page lists Google-hosted and partner-hosted models and mentions no local option. Antigravity’s documentation does describe local models for its Python SDK, through LiteRT or an OpenAI-compatible server such as Ollama, but that page is about the SDK, not the agy command.

If the goal is a Gemini-CLI-style terminal agent on a local model, Qwen Code is the direct route.

Approvals, sandboxing and keeping data on the machine

A local model makes more mistakes with tools than a hosted frontier model, so the approval mode matters. Qwen Code has five: Plan, which is read-only; Ask Permissions, where every edit and shell command needs approval; Auto-Edit; Auto, the default, where a classifier decides; and YOLO. Switch to Ask Permissions, set tools.approvalMode to default, until you trust the model on your codebase.

For isolation, Qwen Code can run inside macOS Seatbelt, whose default profile restricts writes outside the project folder, or in a Docker or Podman container. Our guide to sandboxing agents that execute code explains when each level is enough.

Three settings govern what leaves the machine besides model requests:

{
  "privacy": {
    "usageStatisticsEnabled": false
  },
  "general": {
    "enableAutoUpdate": false
  },
  "model": {
    "catalogRefresh": false
  }
}
  • Usage statistics are enabled by default for all authentication methods and go to an Alibaba Cloud collection endpoint. QWEN_USAGE_STATISTICS_ENABLED=false does the same from the environment and takes precedence.
  • The model catalogue is refreshed from models.dev at most once every 24 hours; disabling usage statistics also stops it.
  • OpenTelemetry export under telemetry.* goes to a destination you configure. Leave telemetry.logPrompts off unless you intend to store prompts.

Gemini CLI has the same privacy.usageStatisticsEnabled setting, also true by default.

How VDF AI fits

VDF AI does not run, host, resell or integrate with Qwen Code, Gemini CLI, Antigravity CLI or LiteLLM. A terminal agent on a local model gives one developer a private workflow. It leaves the model choice, the approval mode and the telemetry switches in a file under each person’s home folder, with no shared record of what the agent read or changed.

VDF Code is VDF AI’s governed coding assistant for organisations that need those decisions made once. Its product page describes deployment in a VPC, in your data centre or fully air-gapped, on Kubernetes, OpenShift or bare metal, with editor extensions for VS Code, JetBrains IDEs, Neovim and Visual Studio. VDF AI Router routes requests across local open-weight models and approved external endpoints under allow and deny lists, and its air-gap mode keeps routing on local models only.

Sources

Repository documentation, model cards and vendor pages consulted on 14 November 2026.


Running coding agents on local models across a team? See how VDF Code handles it on infrastructure you control, or get in touch.

Frequently asked questions

How do I use Qwen Code with a local model?

Serve a model behind an OpenAI-compatible API with Ollama, LM Studio, llama.cpp or vLLM, then add it under modelProviders.openai in the settings.json file in your .qwen folder with an id, a baseUrl ending in /v1 and an envKey that names a placeholder key. Set security.auth.selectedType to openai and model.name to the id, or pick the model with /model. Pin generationConfig.contextWindowSize to the context the server really loaded, and use a model with tool calling.

How do I connect Qwen Code to Ollama?

Pull a model such as qwen3-coder:30b, start Ollama with a larger context, for example OLLAMA_CONTEXT_LENGTH=65536 ollama serve, and add an entry under modelProviders.openai whose baseUrl is the /v1 path on localhost port 11434 and whose envKey points at any placeholder value. For a quick test, export OPENAI_API_KEY, OPENAI_BASE_URL and OPENAI_MODEL instead and run qwen. Ollama has no launcher for Qwen Code, so the configuration is written by hand.

Does Qwen Code work with LM Studio?

Yes. Qwen Code's documentation lists LM Studio next to Ollama and vLLM as a local server reached through the openai provider, with a base URL of localhost port 1234 and the /v1 path. Load the model in LM Studio with enough context first, for example with lms load and the --context-length flag, use the identifier LM Studio serves as the model id, and set contextWindowSize in the entry to the same number so Qwen Code compacts the conversation at the right point.

Can Gemini CLI use a local LLM?

Not natively. Gemini CLI authenticates with a Google sign-in, a Gemini API key or Vertex AI, and its documentation has no Ollama or OpenAI-compatible provider. The only local model it runs itself is a small Gemma model that classifies requests for routing between hosted Gemini models. It does read GOOGLE_GEMINI_BASE_URL, which redirects Gemini API requests to another address, so a proxy that speaks the Gemini API, such as LiteLLM, can forward them to a local model. Google does not document that use.

Can Gemini CLI use Ollama?

Only through a translating proxy. LiteLLM publishes a Gemini CLI tutorial: set GOOGLE_GEMINI_BASE_URL to the proxy address and GEMINI_API_KEY to a LiteLLM key, then use a model_group_alias so that the Gemini model name the CLI requests is routed elsewhere. Pointing that alias at an Ollama model through LiteLLM's Ollama provider gives a local route. It is unsupported by Google, and Gemini CLI's prompts and tools are written for Gemini models, so expect weaker results than with Qwen Code.

Does Qwen Code send data anywhere when it runs on a local model?

Model requests go to the endpoint you configured, but usage statistics are enabled by default for every authentication method and are sent to an Alibaba Cloud collection endpoint. Set privacy.usageStatisticsEnabled to false in settings.json, or set the QWEN_USAGE_STATISTICS_ENABLED environment variable to false, which takes precedence. Disabling it also stops the background refresh of the models.dev catalogue. Automatic update checks are a separate setting, general.enableAutoUpdate. Confirm the result with egress monitoring on locked-down networks.

Filed under
Qwen CodeGemini CLIlocal LLMOllamavLLMAI coding assistants
On-Prem AI

Plan your on-prem AI deployment

Book an architecture call and we will scope a private, on-prem AI deployment for your environment — integrations, hardware, and governance included.

Keep reading