Qwen On-Premise: Choosing a Qwen3.5, 3.6 or 3.8 Model by Size Tier

Qwen on premise means serving Alibaba's Qwen open-weight models on hardware you control. As of September 2026 the family runs from 0.8B to 2.4T parameters, and almost everything below the two newest large models is Apache-2.0. The real decision is the size tier: which dense or mixture-of-experts checkpoint fits your GPUs, context length and tool-calling needs.

Qwen is attractive on premises for a plain reason: few open-weight families cover as many sizes under one permissive license, with official quantized builds and serving instructions for each. That breadth also makes the choice harder. Qwen3.5, Qwen3.6, Qwen3.8 and Qwen3-Coder-Next overlap, and two of the newest checkpoints carry licenses that behave differently from Apache-2.0.

This guide sorts the family into tiers you can match to hardware. Everything here was checked against the Qwen organisation on Hugging Face on 27 September 2026; the family ships often, so confirm before you pin a version. For the wider selection process, see our model selection framework.

The Qwen family as of September 2026

Four release lines matter for enterprise deployment:

• Qwen3.5 (February 2026). Dense models at 0.8B, 2B, 4B, 9B and 27B, and mixture-of-experts models at 35B-A3B, 122B-A10B and 397B-A17B. All are Apache-2.0 and all include a vision encoder.
• Qwen3.6 (April 2026). Two models, 27B dense and 35B-A3B, updated for agentic coding and able to preserve reasoning from earlier turns. Apache-2.0.
• Qwen3.8 (August 2026). Qwen3.8-27B under Apache-2.0, the Qwen3.8-2.4T-A95B flagship, and Qwen3.8-Flash-Next, which Qwen calls an experimental preview of the architecture behind Qwen4.
• Qwen3-Coder-Next (late January 2026). An 80B-total, 3B-active model built for coding agents. Apache-2.0.

The naming encodes the architecture. “35B-A3B” means 35 billion parameters in total, of which about 3 billion are active for each token. A name without an “A” suffix is a dense model.

Dense or mixture of experts

A mixture-of-experts model routes each token through a few expert sub-networks. Qwen3.5-397B-A17B, for example, has 512 experts and activates 10 routed experts plus one shared expert per token. You pay memory for all 397B parameters and compute for about 17B.

That split drives the choice:

• Memory is the constraint. A dense 27B model delivers more quality per gigabyte of GPU memory than a sparse model of similar speed. If one 80 GB card is your budget, the 27B tier is where to start.
• Throughput is the constraint. A 35B-A3B model generates tokens with roughly the compute of a 3B dense model once its weights are resident. For high-volume extraction, classification or routing, it serves more requests per GPU.
• Quality ceiling is the constraint. Only the large MoE tiers (122B-A10B and 397B-A17B) raise capability substantially, and both need multiple GPUs.

All Qwen3.5, 3.6 and 3.8-27B models share a hybrid layout: three Gated DeltaNet layers, which carry a fixed-size state, for every one layer of full attention. Only the full-attention layers grow a KV cache, which changes the long-context arithmetic below.

Licenses differ inside the family

Apache-2.0 covers most checkpoints, but the two newest large models do not use it. We read each license file on 27 September 2026:

Both non-Apache licenses exempt internal use, provided the software, its outputs and its capabilities are not made available to any third party. They define an AI Work Assistant as an independent product built mainly for AI-assisted coding or office productivity. On a plain reading, an assistant used only by your own staff is internal use, while selling a coding assistant or exposing the model to customers through an API is not. Where contractors, subsidiaries or customers can reach the model, ask counsel; our note on license clauses that create work covers the pattern.

Size tiers and the GPUs they need

The table uses repository sizes from the Hugging Face file listing and GPU capacities from NVIDIA’s L40S, H100 and H200 pages. Weight sizes are a floor: KV cache, the vision encoder, activations and runtime overhead come on top.

The last row uses NVIDIA’s DGX B300 total of 2.1 TB of GPU memory. The 122B row shows why “fits” needs care: 127 GB of FP8 weights on one 141 GB H200 leaves about 14 GB, which a few long sessions would exhaust.

KV cache per token follows from each model card’s attention layout. With a 16-bit cache, Qwen3.5-27B stores about 64 KiB per token across its 16 full-attention layers, so one 128K-token session needs about 8 GiB. Qwen3.5-35B-A3B stores about 20 KiB per token, about 2.5 GiB for the same session. Each sequence also holds a fixed DeltaNet state that does not grow with length. On an 80 GB GPU, FP8 Qwen3.8-27B therefore leaves room for roughly four or five concurrent 128K-token sessions after overhead, or many more short ones. The general method is in our GPU sizing arithmetic.

Official quantized builds

Qwen publishes three kinds of quantized artifact, and knowing which exist saves you from trusting an unknown converter:

• FP8. Available for Qwen3.5 from 27B upward, both Qwen3.6 models, all three Qwen3.8 models and Qwen3-Coder-Next. The model cards describe fine-grained FP8 with a block size of 128, and most of them report performance nearly identical to the original.
• GPTQ Int4. Available for Qwen3.5-27B, 35B-A3B, 122B-A10B and 397B-A17B only. The documented vLLM and SGLang commands pass --quantization moe_wna16.
• GGUF. Qwen3-Coder-Next-GGUF ships Q4_K_M (≈48 GB), Q5_K_M (≈57 GB), Q6_K (≈66 GB) and Q8_0 (≈85 GB) files for llama.cpp.

We found no official AWQ builds for Qwen3.5 or later; the most recent official AWQ repositories are for Qwen3. Community AWQ and GGUF conversions exist for most sizes, but each is a separate artifact that needs its own provenance check and evaluation. For choosing between formats, see FP8 versus 4-bit trade-offs.

Long context and tool calling, configured as documented

The model cards give specific settings, and deviating from them causes most of the “Qwen got worse on premises” reports:

• Context. Qwen3.5 and 3.6 run 262,144 tokens natively. Qwen advises keeping at least 128K to preserve thinking quality, and suggests reducing the window if you hit out-of-memory errors. YaRN rope scaling extends them to about one million tokens. Because engines apply static YaRN, set the scaling factor to what you need (the cards give 2.0 for about 524K tokens) and enable it only on deployments that need it. Whether you need that length at all is a separate question, covered in long context versus retrieval.
• Tool calling. The Qwen3.5 and 3.6 cards serve tools with --enable-auto-tool-choice --tool-call-parser qwen3_coder on vLLM, and --tool-call-parser qwen3_coder on SGLang, together with --reasoning-parser qwen3. The Qwen3.8 cards point to the vLLM and SGLang recipes for exact flags.
• Thinking mode. Qwen3.5 and later think by default. Disable it per request with chat_template_kwargs: {"enable_thinking": false}. Qwen3.6 adds a preserve_thinking option. Qwen3.8 turns preserved thinking on by default and adds reasoning_effort with low, medium and xhigh levels.
• Text-only serving. vLLM’s --language-model-only flag skips the vision encoder and frees memory for KV cache when you do not need images.
• Engine versions. The cards ask for recent builds: vLLM 0.19.0 or later for Qwen3.6, and vLLM 0.15.0 or SGLang 0.5.8 or later for Qwen3-Coder-Next.

Coder and vision variants

Qwen3-Coder-Next is the specialist for coding agents: 80B parameters with 3B active, 262,144 tokens of native context, and no thinking mode, so it never emits <think> blocks. Its repository also contains Python tool-parser files for vLLM and SGLang. Review them like any other code in the import pipeline, or use the parser built into your engine.

Vision is not a separate line any more. Every Qwen3.5 and Qwen3.6 model, Qwen3.8-27B and Qwen3.8-Flash-Next are described as causal language models with a vision encoder, and the cards document image and video input. The 2.4T flagship is the exception: its open checkpoint is text-only. Built-in vision removes a model from your fleet, but it costs memory you may not need, which is what the text-only flag above is for.

How VDF AI fits

Most teams end up running more than one Qwen tier: a small or MoE model for volume, a 27B or larger model for harder work. VDF AI Router holds local deployments alongside any external models policy allows. Its SEEMR engine learns from quality, latency, failures and energy which registered model wins for which kind of request, while policy rules (pinned models, allow and deny lists, regulated domains) are applied first and cannot be overridden by the learning layer.

When a new Qwen release appears, the Model Evaluation Suite runs your stored test cases against it inside your deployment and compares it side by side with the incumbent. The upgrade then rests on your own results rather than on the model card.