« GPS, Geotagging, Images and Maps | Main | Search Headlines & Links: November 2, 2006 »
November 2, 2006
Yahoo Slurp Adds Wildcard Support For Robots.txt
The Yahoo Search Blog announced that Yahoo's web crawler, aka Yahoo Slurp, now supports wildcards in the robots.txt file. The two parameters that Yahoo now supports include the "*" and the "$." The * will tell Yahoo to do a "wildcard match a sequence of characters in your URL." The & will tell Yahoo to do a "anchor the match to the end of the URL string." Many more details at the Yahoo Search Blog.
Posted by Barry Schwartz on November 2, 2006 4:04 PM
