A tokenizer splits text into sub-word units before the model sees it; in English a token averages about four characters. Both context windows and API pricing are measured in tokens, so token efficiency directly affects cost and how much you can fit in a prompt.
Different models use different tokenizers, which changes how efficiently they handle code and non-English languages. Open models ship their own tokenizer so you can measure token counts exactly.