Generative AI & LLMs
Tokenization
The process of breaking text into model-readable tokens.
Example
The phrase “AI-powered” is split into pieces the model can process.
Why it matters
It makes varied written material usable by systems that work with a defined set of text pieces.
You'll hear it
“The punctuation changed the token count.”
Can you control it?
Set by the team
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Does splitting text destroy the original wording?
- Not necessarily. Many systems can reconstruct the text from its pieces, although preparation rules can affect details.
- Does the process translate the text?
- No. Breaking writing into pieces is different from changing its language.
- Can identical-looking text produce different results?
- Yes. Hidden spaces or different underlying characters can affect how it is divided.