引導式測驗 · 中等 等級

Robots.txt 和 AI 爬蟲測驗

检查 robots.txt 如何对爬网程序规则进行分组、AI 爬网程序令牌有何不同,以及网站所有者何时需要搜索控制或真正的访问保护。

相關引導路徑機器人TXT和AI爬蟲
問題 1 的 8

A publisher wants ChatGPT search discovery but excludes possible training crawls. Which distinction in OpenAI documentation matters?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. A publisher wants ChatGPT search discovery but excludes possible training crawls. Which distinction in OpenAI documentation matters?
    • OAI-SearchBot and GPTBot have separate controls and purposes
    • One group controls every OpenAI product
    • ChatGPT-User is the search indexing crawler
    • Both preferences require blocking every bot
  2. A site disallows /members/ but the pages remain accessible to anyone with a URL. What went wrong?
    • Disallow only works on images
    • Robots.txt is a crawler request, not authentication
    • The URL must be in a sitemap
    • Robots.txt encrypts pages for compliant bots
  3. What does Google say Google-Extended controls?
    • Whether a URL requires a password
    • All Googlebot access and ranking
    • Certain uses of crawled content for Gemini and Vertex AI without affecting Search inclusion or ranking
    • Whether Chrome renders styles
  4. Why may a robots.txt block fail to remove a page already listed in search?
    • The file changes canonical metadata
    • Disallow sends a removal request
    • A URL must first be de-indexed
    • The crawler can know the URL without fetching its contents
  5. How should a site protect confidential customer records?
    • Disallow the directory but leave it public
    • Enforce authentication and authorization on the server
    • Rename the path
    • Ask compliant crawlers not to visit
  6. Why check current crawler documentation before writing AI-specific groups?
    • Directives are encrypted by vendors
    • All crawlers share a universal token
    • Crawler names and product purposes differ and can change
    • The file detects company goals
  7. An unknown scraper ignores Disallow. What limitation does this show?
    • Robots.txt depends on crawler cooperation
    • The page becomes unindexable
    • All scrapers must publish IP ranges
    • Disallow blocks at the network layer
  8. A request claims a familiar crawler user-agent. What should an operator consider before blocking by that string alone?
    • Every crawler request is harmless
    • The claim proves official identity
    • Robots.txt authenticates the agent
    • User-agent strings can be spoofed, so verify traffic first