salescitation. home

Help center · Glossary · Behind the scenes

What is a web crawler? (in plain English)

Think of it like the meter reader who walks every street, noting what’s at each house.

In plain terms

A program that visits web pages automatically to read them, for a search engine or an AI service. Also called a bot or spider.

Why it matters for a small business

If crawlers can’t reach your pages, search engines and AI assistants can’t read them. Security settings, robots.txt rules or a firewall can turn crawlers away without anyone noticing.

Why ChatGPT isn’t recommending your business →

← All terms in the glossary