15
Apr
Introducing Yahoo! Slurp 3.0
Comments
Yahoo! has announced the launch of its new web crawler, Yahoo! Slurp 3.0. This new crawler is very similar to its previous kin, but is slightly changed to include a few new elements.
1. The crawlers will reside on a new set of IP addresses, but will still reverse DNS lookup to the crawl.yahoo.net domain. So if you are using IP-based rules, make sure to keep an eye out for the new Slurp crawler and add those sets of IP addresses to your database.
2. The new crawlers will return a new user agent, specifically Yahoo! Slurp/3.0. If you set directives for “Slurp” or “Yahoo! Slurp,” those will remain working, but if you specified “Slurp/2.0,” then you may have an issue. Yahoo recommends using User-agent: Slurp in your robots.txt file.
We’ve been in the digital marketing field for over 18 years and worked with hundreds of Australian (and international) businesses to grow their web presence. Specialising in SEO, search ads (PPC), social media, content marketing, email marketing and conversion rate optimisation.
Latest posts by E-Web Marketing (see all)
- Why do most businesses fail at digital marketing? - March 20, 2019
- Want to know your competitors Adwords budget? - November 8, 2017
- The Seven Deadly Sins of Link Building [Infographic] - August 8, 2017
