All activity
Alexander Hjalmarsson
left a comment
Hi, I've published the results gathered from the analysis made earlier from searching through almost 100 million hosts for their robots.txt content looking for security issues[1].
[1] https://www.cdsrc.com/posts/robots-txt-security-measure/
Robots.txt search engine
Most commonly disallowed paths, crawlers, etc
Search engine for robots.txt contents. Indexes the paths and access rules for which enabled you to find hosts that has specific disallow rules and see the most disallowed crawler agents.
Robots.txt search engine
Most commonly disallowed paths, crawlers, etc