Generative AI & LLMs
TF-IDF
Term frequency–inverse document frequency
A way to highlight words that appear often in one document but less often across other documents.
Example
A search system gives a distinctive technical word more weight than a common word.
Why people use it
It helps a search tool give distinctive words more attention than very common ones.
What you'll hear
“That unusual term should matter more in this document.”
What this means for you
Use it as a transparent text signal with known limitations.
Can you control it?
Developer-only
The people building or running the AI choose this setup. An everyday user generally needs their help to change how this part works.
Common questions
- Does TF-IDF capture every semantic relationship?
- No. It counts word patterns rather than fully understanding what the writing means.
- Can a word's weight change when the collection changes?
- Yes. A word that was rare may become common as more documents are added.
- Does it know that two different words mean the same thing?
- Not by itself. It mainly counts the words actually present.