All activity
Alexander Hjalmarssonleft a comment
Hi, I've published the results gathered from the analysis made earlier from searching through almost 100 million hosts for their robots.txt content looking for security issues[1]. [1] https://www.cdsrc.com/posts/robots-txt-security-measure/
Robots.txt search engineMost commonly disallowed paths, crawlers, etc
Search engine for robots.txt contents. Indexes the paths and access rules for which enabled you to find hosts that has specific disallow rules and see the most disallowed crawler agents.
Robots.txt search engineMost commonly disallowed paths, crawlers, etc