Generative AI & LLMs
Token
A small unit of text, such as part of a word, that a language model reads and generates.
Example
A tokenizer may represent “unbelievable” as several smaller pieces rather than one word.
Why it matters
It gives language tools manageable pieces for processing and counting text.
You'll hear it
“This short-looking message uses more tokens than I expected.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Is one token always one word?
- No. A word can use several pieces, and spaces or punctuation can affect the count.
- Do all languages use the same number for equal-length messages?
- No. Counts depend on the language and the particular system's way of dividing text.
- Can token counts affect price?
- Yes. Some services charge according to how many pieces they process or generate.