Generative AI & LLMs
Tokenizer
The component that converts text into tokens a model can process.
Example
Before a model reads a prompt, software converts the text into token identifiers.
Why people use it
Knowing how “Tokenizer” works helps teams choose and configure generative systems more effectively.
What you'll hear
“How does Tokenizer affect the answer the model gives the user?”