Too Cool for Internet Explorer

Friday, January 14, 2005

Not If I Can Help It

This week in Google news

Google remains the favorite of search consumers, but Yahoo and Microsoft are closing the gap.


All well and good, but MSN isn’t going to gain as much ground if people starting doing the same as me and tell MSNbot to not crawl my photo server. The MSNbot hammers my server—much less friendly than any of the other bots. I only see Yahoo and Google pop in every once in a while. I had to block MSN because they were actually taking processor power away from serving my photos to legitimate viewers.

Granted, I run my photos on a 450mhz machine which doesn’t have a lot of power to begin with, but that’s no excuse. MSN shouldn’t hammer any server—no matter how powerful it is—as hard as it was hitting me.

Update: Sometimes, you just gotta hand it to the rare helpful person at an unexpected time. After a short (but polite) rant about MSNbot’s hammering, I actually received a reply with a solution. If you, like me, wish to tighten the reigns of MSNbot, add the following to your robots.txt file:

User-agent: MSNbot
crawl-delay: 120

The 120 value is in seconds and you can change it to any amount of time you desire.

Now then, does anyone have any idea if that is an MSN-only value, or is it supposed to be recognized by any bot that purports to play nice? If the latter, then the User-agent can be changed to * (asterisk) and your “amazing-slow-downer” would be applied to all bots.

» Posted by ALBj at 03:25 PM (ET)
Category: Musings


Sorry, due to comment spam abuse, new comments on this entry are closed until I find time to upgrade Movable Type and enable registration and moderation.