Not all agents (otherwise known as crawlers, bots, robots and spiders) that visit your site will be of benefit. Even the “good” spiders, such as the ones Google sends out to index your site, may visit places that you don’t wish them to.
Malicious spiders, or Web strippers, can cause you a great deal of grief by taking up server resources and increasing your bandwidth usage – this can result in excess bandwidth fees. People use Web stripper applications (also known as offline browsers) to download your entire site. Sometimes their goal is fairly innocent – to go through your site while offline using a locally stored copy. In other circumstances, there may be a much more devious goal – plagiarism or hacking.
How To Ban Spiders
If you notice entries such as Teleport Pro and WebStripper in your traffic reporting, there is something you can do about it – either via a robots.txt file or through meta-tags.
The robots exclusion tag is very simple to implement, but it’s mainly of benefit in keeping search engine spiders out of sensitive areas. Unfortunately, most Web stripping applications ignore it.
The following META tags can be used and should be placed between your <head> and </head> tags:
<META NAME="ROBOTS" CONTENT="NOINDEX">
This will prevent most search engine spiders and some Web strippers from accessing the page.
<META NAME="ROBOTS" CONTENT="NOFOLLOW">
The page will be still be indexed, but any hyperlinks in that page will not be followed by the spider.
The best method is to combine the two:
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
The page will not be indexed and no links will be followed.
The robots.txt file is a more powerful strategy. It is a text file that contains instructions on what to allow/disallow agents and spiders to view and spider on your site. These rules are called The Robots Exclusion Standard.
Most hosts won’t place a robots.txt file in your site when you open an account, so you’ll need to create one in notepad and upload it via FTP to your document root directory. If you are using Microsoft FrontPage, save the file to the root directory of your disk-based site and then upload via FrontPage’s standard HTTP:// publishing function.
Never use a blank robots.txt file as some search engines may see this as an indication that you don’t want your site spidered at all! Have at least one entry in the file and remember to skip a line between entries. Also ensure that the spider/agent that you are banning doesn’t turn out to be a legitimate software browser.
To prevent specific agents and spiders from having any access to your site, put these lines into the robots.txt file:
You must record the name of the agent exactly as it appeared in your traffic reports. For example: WebZip/4.0.
Skip a line between entries. You could do the same to exclude search engine spiders such as Googlebot. The “/” means disallow access to any directory.
You can also prevent access to specific folders:
In this example, the * indicates “all” but please note that the wildcard (*) cannot be used on the Disallow line. Use “/” instead.
Example robots.txt file
If you would like some sort of guide and further examples of a robots.txt file, you can take a look at the one we use on the ThinkHost site.
Our file is by no means complete, but it does contain a number of “idiot” bots that repeatedly attempt to strip our main site. Please be aware that robots.txt will not stop all Web stripping activity as many strippers can fake agent names, but it will help you save on bandwidth.
If you would like to be able to identify the “good” spiders that may visit your site, you can view a listing of the most popular search engines’ robots in our tutorial, Understanding your Web site traffic.
[Michael Bloch, Team ThinkHost]
ThinkHost teams up with Lockergnome!
ThinkHost, a socially responsible and earth-friendly hosting company, wants you to try its service – all Lockergnome visitors can get three free months of Web hosting! This is top quality, full featured hosting powered by the wind and sun – wow! Now that’s different! Get your free green hosting package today! Click here for your special Gnomie code and more information!