A Closer Look at Microsoft’s MSNBot
06/15/04For more than seven years, leaders at Microsoft have hinted at the possible release
of a Microsoft owned search engine platform. While those same seven years have
passed, we have seen MSN, Microsoft's leading user-portal, feature a number of
Not one of which was actually their own technology.
In today's marketplace though, our world is filled with high profile search
engine systems. While Google may dominate the existing landscape - it is obvious
that the folks at Yahoo and Microsoft each have their own dreams well into the
Anyone watching news headlines for the past year or so should be aware that
Yahoo is a serious contender in the search engine industry. Yahoo has been the
most active player in this field having made significant movements in the past
year, most notably with it's acquisition of Inktomi. When someone mentions Microsoft
though - why should we believe that MSN may actually become a major player in
the search engine marketing industry?
Early speculations first indicated that MSN would begin to use its own search
engine platform in the middle of 2005. Back in mid-March however, significant
press circulated around a late-2004 launch. That launch, some say, is about
to happen sooner than we may want to think.
Towards the end of February of this year a new search engine spider began showing
up in server logs everywhere. "MSNBot" had made its first appearance,
and quickly became the topic of discussion. Just as some webmasters became excited
to see those first few traces of MSNBot, others became a bit curious as they
found evidence of the deepest crawl cycles in months by any engine.
When that initial deep crawl began, site owners and marketing managers were
extremely excited and with good reason. Having seen Google make drastic changes
in both updating indexes and algorithmic ranking - any positive news for optimizers
was bound to be the focus of attention and the topic of the hot conversations.
Fast forward just a few months later - and the topic of discussion turned out
to be how abusive MSNBot could appear to be. While Microsoft is certainly no
stranger to negative press - becoming the new kid on the block in this industry
requires a level of responsiveness that many mature organizations cannot afford.
Surprisingly though, this is where one of the largest technology companies
in the world stepped up and catered to the demands of the smalltime webmaster.
It is common to see spiders and crawlers digging through thousands of pages
on a single web site each day. For some webmasters, if this does not happen
- they get worried. For others though, paying for bandwidth and controlling
their server environments becomes are far more important than the traffic potential
a new (unreleased) search engine may bring.
So when webmaster complained that MSNBot was too aggressive, MSN representatives
stepped up to the plate and did the right thing. Over the past month or so,
MSNBot had begun to request pages at a much faster rate than many of us had
seen since Google stopped its monthly updating cycles. Seeing a new spider,
for a new (unreleased) engine doing this - simply did not make sense for many
- and webmasters began taking action against MSNBot's ability to do this. In
typical fashion, the industry responded as webmasters began to speak out against
MSNBot as they would any other negative situation.
In a surprising maneuver though, a MSN Search representative (under the alias
of "msndude") informed webmasters that as of June 11th, MSNBot was
able to review what your preferences were for crawling time. Using the massively
popular Webmaster World forum system, the following discussion had begun:
Controlling a search engine spider through the robots.txt file is nothing new.
But the ability to control the retrieve-rates of a spider is an excellent step
in the right direction.
If you have been concerned about MSNBot's tendencies in working with your site,
I strongly encourage you to review this referenced discussion. Not only will
you learn more about MSNBot and it's methods for finding information - but you
will also become much more in touch with other similar-minded webmasters throughout
In conclusion, let's not forget just how powerful Microsoft has been with technology.
It is clear that MSNBot is a key component to Microsoft and MSN having built
out a large search engine platform. Personally, I would let MSNBot have its
way with every site I have managed
You simply cannot doubt Microsoft's
intentions to succeed.
Looking for a way to add value to your
website visitors' experience? Add these
articles to your site. First Listings offers
a unique news and article datafeed that
allows webmasters to easily link to the
latest industry news and online marketing
tips. Click here
for the code!