Generative AI & LLMs
Token
A small unit of text, such as part of a word, that a language model reads and generates.
Example
A tokenizer may represent “unbelievable” as several smaller pieces rather than one word.
Why people use it
Teams use “Token” when they need to choose and configure generative systems more effectively.
What you'll hear
“How does Token affect the answer the model gives the user?”