Revolutionized Google Crawler Docs: Comprehensive and User-Friendly

Revolutionized Google Crawler Docs: Comprehensive and User-Friendly

Google’s Major Update on Crawler Documentation

Google has completely revamped its Crawler documentation, adding a higher level of detail and topical coverage.

Overview of Changes

A significant restructure has split the single, extensive overview page into three more focused pages:

  • Common Crawlers
  • Special-Case Crawlers
  • User-Triggered Fetchers

This move aims to condense information, making it more accessible and easier to navigate.

Key Updates

Several critical updates have been made:

  • An updated user agent string for GoogleProducer crawler.
  • New content encoding details.
  • A fresh section on technical properties.
Expanded Content Encoding Information

Google’s crawlers now support:

  • gzip
  • deflate
  • Brotli (br)

Each Google user agent now advertises supported content encodings in the Accept-Encoding header.

Goals of the Revamp

The previous overview page had become too comprehensive. Splitting it into sub-topics allows for easier navigation and the inclusion of more detailed information.

Detailed Descriptions of New Pages

Common Crawlers

This section covers widely used crawlers, such as:

  • Googlebot
  • Googlebot Image
  • Googlebot Video
  • Google-StoreBot
  • Google-InspectionTool

These crawlers obey the robots.txt rules and are associated with GoogleBot.

Special-Case Crawlers

These crawlers relate to specific products and respect user agreements:

  • AdSense: Mediapartners-Google
  • AdsBot: AdsBot-Google
  • Google-Safety: Google-Safety

User-Triggered Fetchers

Fetchers in this category are initiated by user actions and generally ignore robots.txt rules, such as:

  • Feedfetcher
  • Google Publisher Center
  • Google Site Verifier

Benefits and Takeaways

The reorganization addresses the issue of an overly comprehensive page by breaking it down into more manageable, topic-specific sections. This makes it easier for users to find the exact information they need and allows for further content expansion.

Conclusion

Splitting the information into three specific pages makes Google’s documentation more user-friendly. This reorganization not only provides detailed insights but also sets the stage for future updates.


For SEO professionals and webmasters, these changes provide a more streamlined and detailed reference, aiding in better search engine optimization practices.