A Closer Look at Microsoft’s MSNBot

06/15/04

In today's marketplace though, our world is filled with high profile search engine systems. While Google may dominate the existing landscape - it is obvious that the folks at Yahoo and Microsoft each have their own dreams well into the development stage.

Anyone watching news headlines for the past year or so should be aware that Yahoo is a serious contender in the search engine industry. Yahoo has been the most active player in this field having made significant movements in the past year, most notably with it's acquisition of Inktomi. When someone mentions Microsoft though - why should we believe that MSN may actually become a major player in the search engine marketing industry?

Early speculations first indicated that MSN would begin to use its own search engine platform in the middle of 2005. Back in mid-March however, significant press circulated around a late-2004 launch. That launch, some say, is about to happen sooner than we may want to think.

Towards the end of February of this year a new search engine spider began showing up in server logs everywhere. "MSNBot" had made its first appearance, and quickly became the topic of discussion. Just as some webmasters became excited to see those first few traces of MSNBot, others became a bit curious as they found evidence of the deepest crawl cycles in months by any engine.

When that initial deep crawl began, site owners and marketing managers were extremely excited and with good reason. Having seen Google make drastic changes in both updating indexes and algorithmic ranking - any positive news for optimizers was bound to be the focus of attention and the topic of the hot conversations.

Fast forward just a few months later - and the topic of discussion turned out to be how abusive MSNBot could appear to be. While Microsoft is certainly no stranger to negative press - becoming the new kid on the block in this industry requires a level of responsiveness that many mature organizations cannot afford.

Surprisingly though, this is where one of the largest technology companies in the world stepped up and catered to the demands of the smalltime webmaster.

It is common to see spiders and crawlers digging through thousands of pages on a single web site each day. For some webmasters, if this does not happen - they get worried. For others though, paying for bandwidth and controlling their server environments becomes are far more important than the traffic potential a new (unreleased) search engine may bring.

So when webmaster complained that MSNBot was too aggressive, MSN representatives stepped up to the plate and did the right thing. Over the past month or so, MSNBot had begun to request pages at a much faster rate than many of us had seen since Google stopped its monthly updating cycles. Seeing a new spider, for a new (unreleased) engine doing this - simply did not make sense for many - and webmasters began taking action against MSNBot's ability to do this. In typical fashion, the industry responded as webmasters began to speak out against MSNBot as they would any other negative situation.

In a surprising maneuver though, a MSN Search representative (under the alias of "msndude") informed webmasters that as of June 11th, MSNBot was able to review what your preferences were for crawling time. Using the massively popular Webmaster World forum system, the following discussion had begun:

http://www.webmasterworld.com/forum97/93.htm

Controlling a search engine spider through the robots.txt file is nothing new. But the ability to control the retrieve-rates of a spider is an excellent step in the right direction.

If you have been concerned about MSNBot's tendencies in working with your site, I strongly encourage you to review this referenced discussion. Not only will you learn more about MSNBot and it's methods for finding information - but you will also become much more in touch with other similar-minded webmasters throughout the world.

In conclusion, let's not forget just how powerful Microsoft has been with technology. It is clear that MSNBot is a key component to Microsoft and MSN having built out a large search engine platform. Personally, I would let MSNBot have its way with every site I have managed… You simply cannot doubt Microsoft's intentions to succeed.