LLM Inference
The process of running a trained model to generate outputs from inputs — the runtime cost centre of AI.
What is LLM Inference?
LLM inference is where compute budget is spent. Latency, throughput, and cost per token depend on model size, quantisation, batching strategy, and hardware. Enterprises that run inference locally (via vLLM, Ollama, llama.cpp) trade cloud convenience for control and predictable economics. See LLM Inference and Where vLLM Fits.
Why it matters for on-premise & regulated AI
Inference is where AI costs and data exposure actually happen — every prompt is a network egress event in a cloud API model. Self-hosted inference (vLLM, llama.cpp, TensorRT-LLM) converts a variable per-token bill into amortized hardware, keeps prompts inside the network, and delivers predictable latency. At sustained enterprise volume, on-prem inference is frequently the cheaper option well before sovereignty is even considered.
Read the full guide: LLM Inference — in-depth article →
Related terms
Putting LLM Inference to work?
VDF AI runs governed AI agents on your own infrastructure — on-premises, sovereign cloud, or air-gapped. Book a working session to map the architecture.
Talk to VDF AI