引導式測驗 · 中等 等級
Robots.txt 和 AI 爬蟲測驗
检查 robots.txt 如何对爬网程序规则进行分组、AI 爬网程序令牌有何不同,以及网站所有者何时需要搜索控制或真正的访问保护。
問題 1 的 8
A publisher wants ChatGPT search discovery but excludes possible training crawls. Which distinction in OpenAI documentation matters?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- A publisher wants ChatGPT search discovery but excludes possible training crawls. Which distinction in OpenAI documentation matters?
- OAI-SearchBot and GPTBot have separate controls and purposes
- One group controls every OpenAI product
- ChatGPT-User is the search indexing crawler
- Both preferences require blocking every bot
- A site disallows /members/ but the pages remain accessible to anyone with a URL. What went wrong?
- Disallow only works on images
- Robots.txt is a crawler request, not authentication
- The URL must be in a sitemap
- Robots.txt encrypts pages for compliant bots
- What does Google say Google-Extended controls?
- Whether a URL requires a password
- All Googlebot access and ranking
- Certain uses of crawled content for Gemini and Vertex AI without affecting Search inclusion or ranking
- Whether Chrome renders styles
- Why may a robots.txt block fail to remove a page already listed in search?
- The file changes canonical metadata
- Disallow sends a removal request
- A URL must first be de-indexed
- The crawler can know the URL without fetching its contents
- How should a site protect confidential customer records?
- Disallow the directory but leave it public
- Enforce authentication and authorization on the server
- Rename the path
- Ask compliant crawlers not to visit
- Why check current crawler documentation before writing AI-specific groups?
- Directives are encrypted by vendors
- All crawlers share a universal token
- Crawler names and product purposes differ and can change
- The file detects company goals
- An unknown scraper ignores Disallow. What limitation does this show?
- Robots.txt depends on crawler cooperation
- The page becomes unindexable
- All scrapers must publish IP ranges
- Disallow blocks at the network layer
- A request claims a familiar crawler user-agent. What should an operator consider before blocking by that string alone?
- Every crawler request is harmless
- The claim proves official identity
- Robots.txt authenticates the agent
- User-agent strings can be spoofed, so verify traffic first