Generative AI & LLMs
Tokenizer
The component that converts text into tokens a model can process.
Example
Before a model reads a prompt, software converts the text into token identifiers.
Why it matters
It prepares writing in a consistent form that a particular AI can process.
You'll hear it
“These two systems split the same sentence differently.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Can switching the tokenizer change the count?
- Yes. Different splitting rules can produce different counts for identical text.
- Does it check whether the writing is true?
- No. Preparing text for processing does not verify any claims in it.
- Can unusual spellings need extra pieces?
- Yes. Unfamiliar words, codes or character combinations may be divided into more pieces.