Tokenization converts text into the units used by a model. A token may represent a whole word, part of a word, punctuation or another text fragment; it is not the same as a word or character. Different tokenizers can split the same sentence differently.
A short common word may be a single token, while a long or rare word, a URL or a line of code can split into several. Text in languages that a tokenizer handles less efficiently can use more tokens for the same meaning, which raises cost and shortens the effective room in the context window. Every large language model has its own tokenizer, so counts differ between providers.
Token counts matter because model context limits and many API prices are measured in tokens. Estimate counts with the tokenizer relevant to the model you use, especially for long prompts, multilingual text and code. Related: Context Window, LLM Inference.