Skip to content

Source

Meta Web Crawlers — documentation

1 item

Meta's own documentation of its web crawlers, their user-agent strings, what each one is used for, and how a site blocks them in robots.txt. Primary for what Meta says its crawlers do and which of them may ignore robots.txt.

developers.facebook.com ↗

Screenshot of Meta's Commercial Terms page, with a notice that the updated terms take effect on October 30, 2026, and the opening paragraph listing the business uses they cover.
Industrymedium signal

Meta's new business terms let it crawl linked websites and use their content for training from October 30

Meta's updated Commercial Terms take effect on October 30, 2026, and add a clause that authorizes it to crawl websites associated with a business account. The content may be used to develop and train Meta's products. Meta calls the update a consolidation of existing terms, and the only alternative the terms offer is to stop using its products for business.

Metaverified