What robots.txt does
robots.txt is a small text file at the top of a website, at example.com/robots.txt. Search engine crawlers read it first to learn which parts of the site they may visit. It also tells them where your sitemap is.
A typical robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlImportant rules
- Blocking isn't hiding. A disallowed page can still appear in Google if other sites link to it. Use a
noindexmeta tag to keep a page out of results, and passwords for private content. - Don't block CSS and JavaScript files; Google needs them to see your pages properly.
- One mistake can hide your whole site:
Disallow: /blocks everything. Double-check before uploading. - Paths are case-sensitive and must start with
/.
Blocking AI training crawlers
Crawlers such as GPTBot, CCBot and Google-Extended collect pages for training AI models. You can block them without affecting normal search results. Well-behaved bots follow robots.txt; it isn't a security measure against those that don't.
Uploading and testing
Upload the file to the top folder of your site (on most hosts, public_html or htdocs), open yoursite.com/robots.txt to check it, and look at the robots.txt report in Google Search Console.