RobotCop

RobotCop [22 Kb]

http://www.robotcop.org/robotcop-src_0.5.tar.gz
http://www.robotcop.org/

“Robotcop is an open source module for web servers which helps webmasters prevent spiders from accessing parts of their sites they have marked off limits. Spiders which read the robots.txt file are held to its rules. If a spider breaks a law in that file, further requests from that spider are intercepted by Robotcop. The webmaster can create trap directories which are marked off limits in the robots.txt file. Spiders which access these trap directories in violation of the robots.txt file are also placed on an intercept list. It includes several interception methods which allows the webmaster to counterattack e-mail address harvesting spiders to trap them, poison their databases, or simply block them. It is a web server module written in C, not a CGI program, which ensures that it does its job very fast. All requests to the site are checked; it even protects requests for other modules, such as PHP. It has a configurable list of known offending spiders which are automatically intercepted on their first request.”