« New SEW Blog Correspondents: Greg Sterling On Local; Detlev Johnson On News & Barry Becomes A Married Chief! | Main | Windows Live Product Search: For Sellers & Buyers »
May 5, 2006
Search Bot Behavior
Ever wonder how spider/bots/crawlers behave? Well, if you did a new analysis "On Bots" was released at http://drunkmenworkhere.org/219. The article has an analysis and visualization of the behavior of search robots. The analysis covers Yahoo Slurp, Googlebot and MSNbot crawling 2 billion pages structured in a binary tree over 1 year. The study was conducted on a single site, so I am not sure how statistically valid it is over all sites on the Web. Just take a look at the overall results to see how much of a hog Yahoo is.
Posted by Barry Schwartz on May 5, 2006 2:56 PM
