If you're new here, you may want to subscribe to my Full RSS feed or sign up for email alerts for FREE!. Thanks for visiting Rhyan dot Net!

Yahoo’s new web-crawler software named Slurp 3.0 has begun indexing the World Wide Web. The new Yahoo’s web-crawler recognizes the same user-agent and robots.txt directives for Yahoo! Slurp.

According to Yahoo’s search blog: Here are some changes, as the new software undergous a phased rollout to their production crawlers in the next several weeks.

a) The crawlers will start crawling from a different and much smaller set of IP addresses, but it’ll still be from the crawl.yahoo.net domain. Any reverse DNS checks to identify our crawler will continue to work. Please note that if you’re using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to reverse DNS-based identification of Yahoo! Slurp if you’re using any other method to avoid this problem. The current set of IPs will disappear from your web logs in the next several weeks.

b) The crawlers will also publish a new user-agent, ‘Yahoo! Slurp/3.0.’ Existing robots.txt directives for ‘Slurp’ or ‘Yahoo! Slurp’ will continue to work, but if you have directives specific to ‘Slurp/2.0,’ they won’t be recognized by the new crawler (though usage of the ‘Slurp/2.0′ user-agent is very rare on the web, so you won’t likely be affected). We recommend specifying the shorter version of: User-agent: Slurp. Check out “How do I prevent my site or certain subdirectories from being crawled?” on our Help page for more details.

The changes will affects main Yahoo! Web search crawlers.