Yahoo! has announced the launch of its new web crawler, Yahoo! Slurp 3.0. This new crawler is very similar to its previous kin, but is slightly changed to include a few new elements.
1. The crawlers will reside on a new set of IP addresses, but will still reverse DNS lookup to the crawl.yahoo.net domain. So if you are using IP-based rules, make sure to keep an eye out for the new Slurp crawler and add those sets of IP addresses to your database.
2. The new crawlers will return a new user agent, specifically Yahoo! Slurp/3.0. If you set directives for “Slurp” or “Yahoo! Slurp,” those will remain working, but if you specified “Slurp/2.0,” then you may have an issue. Yahoo recommends using User-agent: Slurp in your robots.txt file.
Share!