Guide-linked quiz · Medium Level

Robots.txt and AI Crawlers Quiz

Check how robots.txt groups crawler rules, how AI crawler tokens differ, and when site owners need search controls or real access protection.

Related guide pathsRobots Txt And AI Crawlers
Question 1 of 8

A publisher wants ChatGPT search discovery but excludes possible training crawls. Which distinction in OpenAI documentation matters?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. A publisher wants ChatGPT search discovery but excludes possible training crawls. Which distinction in OpenAI documentation matters?
    • OAI-SearchBot and GPTBot have separate controls and purposes
    • One group controls every OpenAI product
    • ChatGPT-User is the search indexing crawler
    • Both preferences require blocking every bot
  2. A site disallows /members/ but the pages remain accessible to anyone with a URL. What went wrong?
    • Disallow only works on images
    • Robots.txt is a crawler request, not authentication
    • The URL must be in a sitemap
    • Robots.txt encrypts pages for compliant bots
  3. What does Google say Google-Extended controls?
    • Whether a URL requires a password
    • All Googlebot access and ranking
    • Certain uses of crawled content for Gemini and Vertex AI without affecting Search inclusion or ranking
    • Whether Chrome renders styles
  4. Why may a robots.txt block fail to remove a page already listed in search?
    • The file changes canonical metadata
    • Disallow sends a removal request
    • A URL must first be de-indexed
    • The crawler can know the URL without fetching its contents
  5. How should a site protect confidential customer records?
    • Disallow the directory but leave it public
    • Enforce authentication and authorization on the server
    • Rename the path
    • Ask compliant crawlers not to visit
  6. Why check current crawler documentation before writing AI-specific groups?
    • Directives are encrypted by vendors
    • All crawlers share a universal token
    • Crawler names and product purposes differ and can change
    • The file detects company goals
  7. An unknown scraper ignores Disallow. What limitation does this show?
    • Robots.txt depends on crawler cooperation
    • The page becomes unindexable
    • All scrapers must publish IP ranges
    • Disallow blocks at the network layer
  8. A request claims a familiar crawler user-agent. What should an operator consider before blocking by that string alone?
    • Every crawler request is harmless
    • The claim proves official identity
    • Robots.txt authenticates the agent
    • User-agent strings can be spoofed, so verify traffic first

Keep testing yourself

More quizzes picked for your level.

View all quizzes