WebmasterWorld Bans Search Engine Bots from Crawling
- 0
- Add a Comment
Sure, not all bots are bad. Yet in the case of bogging down a server, the desire to ban them can certainly make a who lot of sense!
Danny reports that WebmasterWorld Bans Spiders From Crawling where he points to a thread Brett started which he named lets try this for a month or three…
Brett explains that the bots, mostly from unauthorized sources are taking a toll on the server. He explains; “We have pushed the limits of page delivery, banning, ip based, agent based, and down right cloaking to avoid the rogue bots - but it is becoming an increasingly difficult problem to control.” [Read the rest]
Tags: bots, webmasterworld, spiders
