Guide-linked quiz · Medium Level
Robots.txt and AI Crawlers Quiz
Check how robots.txt groups crawler rules, how AI crawler tokens differ, and when site owners need search controls or real access protection.
Question 1 of 8
A publisher wants ChatGPT search discovery but excludes possible training crawls. Which distinction in OpenAI documentation matters?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- A publisher wants ChatGPT search discovery but excludes possible training crawls. Which distinction in OpenAI documentation matters?
- OAI-SearchBot and GPTBot have separate controls and purposes
- One group controls every OpenAI product
- ChatGPT-User is the search indexing crawler
- Both preferences require blocking every bot
- A site disallows /members/ but the pages remain accessible to anyone with a URL. What went wrong?
- Disallow only works on images
- Robots.txt is a crawler request, not authentication
- The URL must be in a sitemap
- Robots.txt encrypts pages for compliant bots
- What does Google say Google-Extended controls?
- Whether a URL requires a password
- All Googlebot access and ranking
- Certain uses of crawled content for Gemini and Vertex AI without affecting Search inclusion or ranking
- Whether Chrome renders styles
- Why may a robots.txt block fail to remove a page already listed in search?
- The file changes canonical metadata
- Disallow sends a removal request
- A URL must first be de-indexed
- The crawler can know the URL without fetching its contents
- How should a site protect confidential customer records?
- Disallow the directory but leave it public
- Enforce authentication and authorization on the server
- Rename the path
- Ask compliant crawlers not to visit
- Why check current crawler documentation before writing AI-specific groups?
- Directives are encrypted by vendors
- All crawlers share a universal token
- Crawler names and product purposes differ and can change
- The file detects company goals
- An unknown scraper ignores Disallow. What limitation does this show?
- Robots.txt depends on crawler cooperation
- The page becomes unindexable
- All scrapers must publish IP ranges
- Disallow blocks at the network layer
- A request claims a familiar crawler user-agent. What should an operator consider before blocking by that string alone?
- Every crawler request is harmless
- The claim proves official identity
- Robots.txt authenticates the agent
- User-agent strings can be spoofed, so verify traffic first
Keep testing yourself
More quizzes picked for your level.