← All explainers

robots.txt controls crawling

Updated: 0 views

It tells compliant crawlers which URLs they may access.

Not automatic removal

Google may show a known URL without crawling it. Crawl blocking is not index removal.

Privacy is separate

Private areas need access control. Disallow is public and cannot stop non-compliant clients.

FinFly practice

Check image and stylesheet access. A crawler must be able to see a noindex directive to process it.

Primary sources

The explainer is checked against these institutional and industry sources. Links open the original material.

All explainers