salescitation. home

Help center · Glossary · Behind the scenes

What is robots.txt? (in plain English)

Think of it like a sign on your gate saying who’s welcome where. Polite visitors obey it.

In plain terms

A small file on your website that tells crawlers which parts they may visit. Well-behaved crawlers follow it; it isn’t a lock.

Why it matters for a small business

One wrong line in robots.txt can hide your whole site from search engines or AI crawlers. It takes a minute to look: open yourbusiness.com/robots.txt in a browser.

Why ChatGPT isn’t recommending your business →

← All terms in the glossary