We are living in an age where robots and spiders are crawling all over your Web site. No, this isn’t a tag line from an old 1950 horror movie, this is the way things are. Don’t be frightened, though. The fact that you have robots and spiders on your Web site is a good thing. A very good thing if you care about your own success online. How can you make the most out of the robots and spiders? It all starts with a little file called “robots.txt.”

Before I get into what the robots.txt file is all about, there is something I have to cover. If you have been around the proverbial Web master block a few times, you have heard about search engine spiders. They are small “robots” that search engines send out across the Internet to look for content. Just about every major search engine uses them.

Now let us start with what it is. The robots.txt file is a small text file that sits in your root directory. When search engines send out spiders to roam the Internet looking for content to pick up, they read the robots.txt file first. Think of it as your way to talk directly to the search engines.

This is how your Web site ends up on a search engine (like Google). When you submit your Web site to a search engine, you are putting your domain on a list of Web sites for it to spider over. Now which is best? Is it better for the search engine spider to find you by itself or with you submitting yourself ot the search engine? There is debate for both sides, so I will not get too deep into that.

[Read the rest of Chat With Search Engine Spiders!]