Free tool. Test crawler access.

Test your robots.txt

Check whether search engines can access your pages. Enter a website URL and a page path to test against the robots.txt rules.