What Is a Token?
A token is the basic unit of text an AI model processes — roughly, but not exactly, a word. Common short words are often a single token; longer or less common words may be split into several tokens (a word like "tokenization" itself might become two or three tokens). Punctuation and spaces typically count as tokens too.
Why Tokenization Exists
Models don't process raw text directly — text first gets broken down (tokenized) into a sequence of numerical IDs the model can actually work with mathematically. Breaking text into sub-word pieces rather than whole words lets a model handle rare words, typos, and multiple languages more flexibly than a strict whole-word approach would.
Why It Affects Pricing
Most AI API pricing is measured per token (often per million tokens), for both the text you send (input tokens) and the text the model generates back (output tokens). This is why the exact same request can cost a different amount depending on how efficiently it's tokenized, and why concise prompts and responses are generally cheaper than verbose ones.
Why It Affects Context Limits
A model's context window — the maximum amount of text it can consider at once — is measured in tokens, not words or characters. This is why a long document might unexpectedly exceed a model's limit even if it doesn't look that long by word count, since some content tokenizes less efficiently than others.
Related Pages
Frequently Asked
Is one token always one word?
No — common short words are often a single token, but longer, rarer, or non-English words are frequently split into multiple tokens.
Why does my API bill vary for similar-length requests?
Because token count, not character or word count, determines cost, and different text tokenizes with different efficiency depending on vocabulary and language.
How can I estimate my token usage?
See our Token Calculator for a rough, practical estimate based on your text.
Does every AI model use the same tokenization method?
No — different providers and model families often use their own tokenization scheme, which is part of why token counts for the same text can vary slightly between models.