Google’s Major Update on Crawler Documentation
Google has completely revamped its Crawler documentation, adding a higher level of detail and topical coverage.
Overview of Changes
A significant restructure has split the single, extensive overview page into three more focused pages:
- Common Crawlers
- Special-Case Crawlers
- User-Triggered Fetchers
This move aims to condense information, making it more accessible and easier to navigate.
Key Updates
Several critical updates have been made:
- An updated user agent string for GoogleProducer crawler.
- New content encoding details.
- A fresh section on technical properties.
Expanded Content Encoding Information
Google’s crawlers now support:
gzip
deflate
Brotli (br)
Each Google user agent now advertises supported content encodings in the Accept-Encoding
header.
Goals of the Revamp
The previous overview page had become too comprehensive. Splitting it into sub-topics allows for easier navigation and the inclusion of more detailed information.
Detailed Descriptions of New Pages
Common Crawlers
This section covers widely used crawlers, such as:
- Googlebot
- Googlebot Image
- Googlebot Video
- Google-StoreBot
- Google-InspectionTool
These crawlers obey the robots.txt rules and are associated with GoogleBot.
Special-Case Crawlers
These crawlers relate to specific products and respect user agreements:
- AdSense:
Mediapartners-Google
- AdsBot:
AdsBot-Google
- Google-Safety:
Google-Safety
User-Triggered Fetchers
Fetchers in this category are initiated by user actions and generally ignore robots.txt rules, such as:
- Feedfetcher
- Google Publisher Center
- Google Site Verifier
Benefits and Takeaways
The reorganization addresses the issue of an overly comprehensive page by breaking it down into more manageable, topic-specific sections. This makes it easier for users to find the exact information they need and allows for further content expansion.
Conclusion
Splitting the information into three specific pages makes Google’s documentation more user-friendly. This reorganization not only provides detailed insights but also sets the stage for future updates.
For SEO professionals and webmasters, these changes provide a more streamlined and detailed reference, aiding in better search engine optimization practices.