1 Crawlers and rules
Each crawler follows only the most specific group that names it. A group for a named crawler does not inherit the rules of the * group. Crawlers differ in purpose; check each provider's documentation.
2 Sitemaps
Optional. Enter each sitemap as a full URL, for example https://example.com/sitemap.xml. Only the format is checked, not whether the sitemap exists.
3 Your robots.txt
Reset returns to the default file and cannot be undone.
Checks
About robots.txt
4 Test a URL
This is a local simulation. It does not test your live website.
For example /private/page.html or https://example.com/blog/?page=2.