A token is the unit a language model splits text into when it reads and writes. A token can be a whole word, part of a word or a single character.
Short, common English words are often a single token, while long Danish compound words are usually split into several. The model works token by token: it calculates which token is most likely to follow the previous ones and writes its way to an answer.
Tokens are also the unit used for pricing and limits. Many AI services charge per token, and the context window sets how many tokens the model can work with at once.
In practice
As an ordinary user, you do not need to count tokens. But they explain why very long documents can be cut short, and why the model sometimes misspells names and specialist terms it has rarely seen.
The concepts in this glossary run through my AI courses and workshops, where we work with them on your own tasks.