Crawler Permission Checker

Inspect a URL's robots.txt, X-Robots-Tag and crawl directives — get a Scrapability Score from 0 (locked down) to 100 (wide open).