Kimi K3 On-Premises: What Serving a 2.8-Trillion-Parameter Open Model Really Takes

Kimi K3 on premises means running Moonshot AI's open-weight 2.8-trillion-parameter mixture-of-experts model on your own GPUs. The Kimi K3 License permits self-hosting and waives its commercial conditions for internal use. The hard part is capacity: the checkpoint is about 1.56 TB, more than one 8-GPU H200 or B200 node can hold.

Kimi K3’s full weights are genuinely downloadable, which makes it tempting to treat as the next model to add to an on-premises fleet. At this size, though, the deployment is an infrastructure project in its own right: node counts, interconnect, storage throughput and failure domains all change.

This guide covers what an architect needs before committing: what was published, what the license asks of you, the memory arithmetic, and what multi-node serving involves. Facts were checked on Hugging Face, in Moonshot’s repositories and in the vLLM recipe on 27 September 2026.

What Moonshot published

Moonshot AI made the Kimi-K3 repository public on Hugging Face on 27 July 2026, the date of its first public commit. Its GitHub repository, created the same day, hosts the technical report. The model card lists:

Two details matter operationally. Moonshot applied quantization-aware training in this format from the SFT stage onward, so the published MXFP4 checkpoint is the intended serving precision rather than a conversion you would need to validate separately. And Kimi K3 always reasons. The model card says thinking is always on, with effort set through reasoning_effort (low, high or max, default max).

The Kimi K3 License in plain terms

The Kimi K3 License reads like MIT for most of its length. It grants the right to use, copy, modify, distribute, deploy and fine-tune the software, which it defines to include the weights. Then it adds three things:

1. A Model-as-a-Service condition. Offering a third party inference or fine-tuning with meaningful control over inputs, parameters or training data counts as Model-as-a-Service. If you or any affiliate run such a business and your group’s revenue exceeds US$20 million over any 12 consecutive months, you need a separate agreement with Moonshot AI before any commercial use. End-user products that embed the model inside specific features, and plain relaying of requests to models hosted elsewhere, fall outside the definition.
2. A naming condition. Commercial products or services with more than 100 million monthly active users, or more than US$20 million in monthly revenue, must display “Kimi K3” prominently in the interface.
3. Two exemptions. Neither condition applies to internal use that does not make the software, its outputs or its capabilities available to third parties, or to use through Moonshot’s official products or certified inference partners.

For a typical enterprise running K3 for its own staff, the internal-use exemption is the clause that matters. The trigger to watch is group structure: a large organisation with any API or cloud business line can meet the Model-as-a-Service test through an affiliate. Compare this with Kimi K2.6, whose Modified MIT License adds only the naming requirement. Our note on open-weight license clauses covers the wider review.

Memory arithmetic per node

A mixture-of-experts model computes with its active parameters but must hold all of them. For K3 that means 1.56 TB of weights before a single request is served.

Capacities come from NVIDIA’s H200, DGX B200 and DGX B300 pages. The GB300 row divides the 20 TB that NVIDIA lists for 72 GPUs in a GB300 NVL72 rack, and the MI355X figure is from the vLLM recipe for DeepSeek V4. The two-node H200 row is arithmetic only: vLLM’s K3 recipe does not list H200 or B200 as tested targets.

The KV cache is modest by comparison, because only 24 of the 93 layers keep a per-token cache. Each Gated MLA layer stores a compressed latent of 576 values per token (a 512-wide KV projection plus a 64-wide rotary component, per config.json). With a 16-bit cache that is about 27 KiB per token across all 24 layers, or about 27 GiB for a one-million-token session. vLLM’s recipe uses an FP8 KV cache, which halves this to about 13.5 GiB per million-token session, or about 1.7 GiB for a 128K-token session. The 69 linear-attention layers add a fixed state per sequence that does not grow with length.

On a DGX B300, the roughly 0.54 TB left after weights would hold the FP8 cache for about 37 full one-million-token sessions, or about 300 sessions of 128K tokens, if nothing else needed memory. Activations, the vision encoder, CUDA graphs and runtime buffers all come out of the same pool, so the real ceiling is lower and has to be measured under load. The general sizing method is in our GPU sizing arithmetic.

What multi-node serving involves

Moonshot’s model card recommends vLLM, SGLang and TokenSpeed. The vLLM recipe is the most detailed public guide, and it states plainly that production traffic should run on multiple nodes. What that means in practice:

• Parallelism layout. The recipe offers tensor parallel on one node (TP8), tensor plus expert parallel across 16 GPUs (TEP16), tensor plus pipeline parallel across two nodes (TP8×PP2) and disaggregated prefill and decode. It labels these profiles pre-release, so expect flags to change between vLLM versions.
• Interconnect. Expert parallelism across nodes sends tokens between GPUs on every MoE layer. The recipe selects different all-to-all backends for RDMA and for cross-node NVLink. Your network fabric becomes part of the model’s latency.
• Software floor. vLLM 0.29.0 or later with an NVIDIA r580 or newer driver; the prebuilt wheels target CUDA 13, and the recipe describes a self-build path for CUDA 12.9 hosts. AMD MI355X deployments use a ROCm nightly image. Plan the upgrade path for drivers and engines before the hardware arrives.
• Storage and cold start. Each replica has to load 1.56 TB of weights, which the recipe assumes come from shared storage. Storage throughput decides how long a restarted node stays out of rotation, and a rolling upgrade needs spare capacity equal to one full replica.
• Failure domains. When one model instance spans several nodes, losing any node takes the whole replica down. Size for N+1 replicas, or accept a fallback model during incidents.
• Conversation handling. K3 was trained in preserved-thinking mode. For multi-turn and tool-calling conversations, the model card says the full assistant message, including reasoning_content and tool_calls, must be passed back unchanged. Any gateway or client that strips reasoning between turns will degrade results.

The recipe’s launch command uses the kimi_k3 tool-call and reasoning parsers, prefix caching and a maximum model length of 1,048,576 tokens. It also advises adjusting that maximum to the scenario, so set it to what your workloads actually need.

Remote code and provenance

Unlike checkpoints that load through built-in engine code alone, the Kimi-K3 repository ships Python: modeling_kimi_k3.py, configuration_kimi_k3.py, tokenization_kimi.py and several processor files. Its config.json maps the Transformers auto classes to those files, and the recipe’s launch command passes --trust-remote-code. That flag lets repository code execute on your servers.

Treat that code as part of the release. Pin the repository commit, review the Python files at that commit, verify the SHA-256 digest of each of the 96 shards against the Hub’s file listing, and re-review whenever the commit changes. Serve from a segment without outbound access. The same discipline applies to any large open model; our DeepSeek provenance checklist walks through it step by step.

When a smaller model is the better call

Few enterprise workloads need a 2.8T model on every request. Before committing to multi-node capacity, price the alternatives:

• Kimi K2.6. 1T parameters with 32B active, a 256K context window and a checkpoint of about 595 GB. By raw capacity it fits in one 8-GPU H200 node with around 530 GB to spare, under a license that asks only for naming at very large scale.
• Other open families. DeepSeek’s Flash checkpoints and the mid and large Qwen size tiers run on a single node or a fraction of one.
• A routed mix. Serve most requests from a single-node model and reserve a K3 replica for the long-horizon coding or research tasks where your evaluation shows it wins.

If you do buy for K3, the capacity decision itself deserves the same rigour as the model choice; see buying, leasing or colocating GPUs.

How VDF AI fits

A model this expensive to serve should only receive the requests that need it. VDF AI Router registers on-premises deployments of different sizes in one catalog and applies policy first: pinned models, allow and deny lists and regulated-domain approvals. Its SEEMR engine then learns from quality, latency, failures and energy where each registered model earns its cost.

Each routing decision carries a reason, ordered failover candidates and per-model scores, and fallback models set per workflow step keep an answer path open while the K3 replica is draining for an upgrade. The case for keeping a multi-node K3 deployment then rests on recorded outcomes rather than on launch benchmarks.