robots.txt Detection

Check whether a website has robots.txt. After configuration, this extension can detect whether other paths exist.