What Is a Token?
A token is the basic unit of text that AI language models process. Tokens are not words — they are chunks of characters determined by the model's tokenizer. Common English words are usually one token; rare words, code symbols, and non-Latin scripts often split into several tokens.
Token counts matter because API costs are billed per token, context windows are measured in tokens, and non-English text typically costs 2–3× more tokens than equivalent English. → What is a token?
How This Counter Works
This tool uses tiktoken (GPT-4o's tokenizer, o200k_base) to count tokens exactly. For other models like Claude and Llama, each provider uses a proprietary tokenizer — so results will differ, but are close enough for budgeting and planning.
Multilingual note: English is the most token-efficient language because tokenizers are trained primarily on English text. The same sentence in Ukrainian costs roughly 2× more tokens; Arabic and Japanese are similar.
Common Use Cases
- Prompt optimisation — trim system prompts before deploying to production
- Cost estimation — calculate API spend before scaling to millions of calls
- Context window planning — ensure your prompt + documents fit within the model limit
- Multilingual budgeting — measure actual token costs for non-English content
- RAG chunking — size document chunks to fit retrieval budgets
Frequently Asked Questions
What is a token in AI?
A token is a chunk of text — typically 3–4 characters for English. The word "token" is one token; "tokenization" may split into two. Punctuation and spaces are often separate tokens. Models process tokens, not words or characters.
How many tokens is 1000 words?
For standard English prose, roughly 1,300–1,500 tokens. A rough rule: 1 word ≈ 1.3 tokens. Code and non-English text will differ significantly.
Does this count Claude or Gemini tokens accurately?
Not exactly — Claude, Gemini, Llama, and other models each use their own proprietary tokenizers that aren't publicly available. This tool uses tiktoken (GPT-4o's tokenizer), which gives a close-enough estimate for planning and budgeting across most modern LLMs. For exact counts, use each provider's official tokenizer tool.
Why does non-English text cost more tokens?
Tokenizers are trained primarily on English text, so English words map to single tokens efficiently. Characters from other scripts (Cyrillic, Arabic, CJK) are less common in training data and split into more sub-word tokens. The same idea in Ukrainian typically costs ~2× more tokens than in English.
Does this tool send my text to any server?
No. Tokenization runs entirely in your browser using the tiktoken library compiled to WebAssembly. Your text never leaves your device. No account, no API key, works offline after first load.
How large are modern LLM context windows?
Context windows vary widely: GPT-4o supports 128K tokens, Claude 3.7 Sonnet supports 200K, and Gemini 2.0 Flash supports 1M tokens. Check each provider's current documentation — limits grow frequently. 128K tokens is roughly 96,000 words or ~300 pages of text.