Robots.txt and AI Crawlers

Robots.txt is a plain-text file served at a site's top-level path, usually example.com/robots.txt. It groups a user-agent token with directives such as Allow and Disallow. These directives ask a crawler which paths it may fetch. Matching syntax, precedence, caching, and supported extensions can vary, so consult the current documentation for the crawler you intend to address.

“AI crawlers” are not one category. Operators may use distinct crawlers for search discovery, model training, or user-triggered retrieval. OpenAI documents separate OAI-SearchBot and GPTBot controls: one is for search features, while the other concerns crawling that may be used for training. Google documents Google-Extended as a robots.txt token for certain Gemini and Vertex AI training and grounding uses, and says it does not affect Google Search inclusion or ranking. Product behavior can change, so check current primary documentation before configuration.

Robots.txt is a request, not a lock. The file is public. It does not authenticate visitors, prevent a person from opening a URL, remove an already indexed page, or force an unknown scraper to comply. Never put secrets in a path and rely on Disallow. Use authentication and server-side authorization for private material. If unwanted traffic ignores a rule, rate limits or network controls may help, but verify traffic first because user-agent strings can be spoofed.

Start by listing the outcomes you want: search visibility, AI search discovery, possible training exclusion, or reduced load on particular paths. Identify each operator's documented token, place rules in separate groups, and check the file at its public URL. Test representative paths with available crawler tools, then inspect server logs for verified requests. Keep a dated copy so you can reverse a mistaken rule. Review the policy when product names or site paths change.