Tokenization is the process of breaking text into tokens — the small units, usually pieces of words, that a language model actually reads and generates. Modern LLMs use subword tokenization, so a common word may be one token while a rare word is split into several. Tokens matter in practice because they are the unit of pricing, the unit that fills the context window, and a source of subtle model behaviors.
Key takeaways
- LLMs do not process characters or words directly — they process tokens, typically subword fragments.
- A rough rule of thumb in English is ~4 characters or 0.75 words per token, but it varies by language and content.
- Tokens are the unit of cost and context: API pricing and context-window limits are both measured in tokens.
- Tokenization quirks explain real behaviors — why models miscount letters, struggle with some numbers, and handle non-English text less efficiently.
What tokenization is
Tokenization is the first thing that happens to your prompt. Before a model can process text, a tokenizer converts it into a sequence of tokens, and each token is mapped to a numeric ID that indexes into the model’s vocabulary. The model never sees your raw characters — it sees these token IDs, which it then turns into embeddings.
Most modern LLMs use subword tokenization (schemes like Byte-Pair Encoding). Frequent words become a single token; rarer or compound words are split into meaningful pieces. This strikes a balance: a purely word-level vocabulary would be enormous and would choke on unseen words, while a purely character-level scheme would make sequences far too long. Subwords keep the vocabulary manageable while handling any input.
Why tokens govern cost and context
Tokens are the currency of LLMs. When you pay per token or reason about throughput, you are counting tokens, not words or characters. A long document might be tens of thousands of tokens; understanding the count is essential for estimating cost and for staying within limits.
Tokens also define the context window — the maximum number of tokens a model can consider at once, spanning both your prompt and its response. If a document plus its instructions exceed that budget, something must be truncated or retrieved selectively. This is why context engineering and retrieval exist: to fit the right tokens into a finite window.
The quirks tokenization explains
Many puzzling model behaviors trace back to tokenization. Because a model sees tokens rather than letters, it can struggle to count the letters in a word or to reverse a string — the individual characters are not cleanly exposed to it. Numbers can tokenize inconsistently, which contributes to arithmetic slips. These are not reasoning failures so much as artifacts of how the input is chopped up.
Tokenization also has real equity and cost implications across languages. Tokenizers are usually optimized for English, so the same sentence in another language — especially one with a non-Latin script — can require substantially more tokens. That means higher cost and less effective context for those languages, an important consideration for global deployments.
Practical implications for enterprises
For anyone deploying LLMs, tokenization is a planning tool. Estimating token counts for your typical documents lets you size context windows, forecast cost, and choose models sensibly. It also informs chunking strategy for retrieval: splitting documents along sensible token boundaries preserves meaning and improves relevance.
When you self-host models, the tokenizer ships with the model and runs entirely within your environment — there is no external call just to count or split text. That keeps even the earliest stage of processing inside your governed boundary, which matters when the input itself is sensitive.
From concept to a governed, on-premise reality
On the VDF AI platform, tokenization happens locally as part of each self-hosted model, so sensitive text is split and processed entirely inside your perimeter — never sent to an external tokenizer or API.
Because tokens drive cost, VDF AI’s flat-rate model and router shield you from per-token cloud pricing: verbose prompts and non-English content do not translate into surprise bills, and the router keeps token usage efficient by matching requests to right-sized models.
Frequently asked questions
What is a token in an LLM?
A token is the unit of text an LLM actually processes — usually a subword fragment, sometimes a whole short word. Text is converted into a sequence of tokens before the model reads it, and the model generates its output one token at a time.
How many words are in a token?
As a rough English rule of thumb, one token is about four characters or three-quarters of a word, so 100 tokens is roughly 75 words. The exact ratio varies with the content and especially with the language.
Why do tokens matter for cost?
Most LLM pricing and all context limits are measured in tokens. Both your prompt and the model’s response consume tokens, so understanding token counts is essential for estimating cost and staying within a model’s context window.
Why do LLMs miscount letters in a word?
Because the model sees tokens, not individual characters. A word may be a single token, so the letters inside it are not cleanly exposed to the model. This tokenization artifact — not a reasoning flaw — explains many letter-counting and spelling quirks.
Does tokenization affect non-English languages?
Yes. Tokenizers are typically optimized for English, so text in other languages — particularly non-Latin scripts — often needs more tokens for the same meaning. That increases cost and consumes context faster, which is a real consideration for multilingual deployments.
Put these concepts to work on infrastructure you control.
VDF AI runs governed agents, private retrieval, and model routing inside your own cloud, data center, or air-gapped network. Book a walkthrough mapped to your stack.