Skip to content
LATHICAI Glossary

RAG & Search

AI crawler

Software that visits web pages to collect or find information for an AI-related purpose.

Example

A publisher sees requests from a crawler associated with an AI service.

Why people use it

It helps services gather web material for search, reference or other AI-related uses.

What you'll hear

“Which crawler is visiting our pages?”

What this means for you

Check the specific crawler's documented purpose and available site controls.

Can you control it?

Sometimes

Sometimes. Your choices depend on the tool and your access. The settings available to an everyday user may differ from those available to the people running it.

Common questions

Does every AI crawler use collected pages for training?
No. Crawlers may serve different purposes, including search and finding information.
Can a crawler miss changes on a website?
Yes. The stored version can become outdated between visits.
Does a crawler visit guarantee inclusion in an answer?
No. Collecting a page does not mean the service will select or cite it later.

Related terms

Still have questions?

Up to 500 characters.

Ask LATHIC about AI. Relevant glossary entries may be included.

Your question, the glossary entries it matches, and a rotating pseudonymous identifier go to Microsoft Azure’s OpenAI service through Vercel AI Gateway to generate an answer. Zero retention and no training are required of the provider, and LATHIC does not save your question or answer. Privacy Notice