Subscribe - Search Engine Watch

Subscribe to Newsletters

Sponsored by:

May 6, 2008

SEW Experts: Conducting a Redirect Audit on Your Web Site - Part 2

Knowing which pages on your site are returning errors, and which ones are being redirected can help you pinpoint issues to address. In today's Organic Search Engine Optimization column, "Conducting a Redirect Audit on Your Web Site - Part 2," Mark Jackson shows how performing a redirect audit can help you get to the bottom of those problems.

Posted by Kevin Newcomb at 12:00 AM | Permalink

April 29, 2008

SEW Experts: Conducting a Redirect Audit on Your Web Site

A redirect audit looks at the server redirects that are happening on your site, and which sites are sending visitors to links on your site that are being redirected. It also looks at 404 errors (file not found), as well as other server status codes appearing in your site's log files. In today's Organic Search Engine Optimization column, "Conducting a Redirect Audit on Your Web Site," Mark Jackson shows you how a redirect audit can take care of many issues that search engines might be having with your Web site, and may help you recover visitors you may be losing due to technical issues.

Posted by Kevin Newcomb at 12:00 AM | Permalink

March 18, 2008

SEW Experts: To Rewrite or Not to Rewrite...That is the Question

While URL rewriting can have SEO benefits, it can also cause more SEO problems if it's done hastily, or incorrectly. In today's Organic Search Engine Optimization column, "Dynamic URLs: To Rewrite or Not to Rewrite...That is the Question," Mark Jackson outlines the pros and cons to consider before making a change.

Posted by Kevin Newcomb at 12:00 AM | Permalink

February 1, 2008

SEW Experts: Rewriting URLs: SEO for CMS, E-Commerce, and Dynamic Sites

There's as much confusion and controversy surrounding URL rewriting as Darwin's theory of evolution. In today's SEM Crossfire column, "Rewriting URLs: SEO for CMS, E-Commerce, and Dynamic Sites," Chris Boggs says that just as Darwin's "survival of the fittest" means species most suited for their environment will adapt and survive, URLs need to adapt too.

Posted by Kevin Newcomb at 12:00 AM | Permalink

January 29, 2008

SEW Experts: SEO Millionaire: Who Wants to Be One? - Part 2

Last week, we challenged SEOs to identify which of the big three travel sites has a canonical issue? In today's au Natural column, "SEO Millionaire: Who Wants to Be One? - Part 2," Mark Jackson provides the answer, and responds to voluminous reader response.

Posted by Kevin Newcomb at 12:00 AM | Permalink

December 4, 2007

SEW Experts: Choosing the Best Domains for Search Engine Visibility

What's in a name? Search engine visibility starts with buying the best domains. In today's au Natural column, "How to Choose the Best Domains for Search Engine Visibility," Mark Jackson explains why choosing the right domains, and creating a domain redirect strategy, can be a valuable move in your SEO plan.

Posted by Kevin Newcomb at 12:00 AM | Permalink

February 19, 2007

The 301 Redirect Headache

Moving Web pages that have been indexed by search engines to a new URL via a 301 permanent redirect can cause serious dips in traffic once the search engines discover that the old page has moved. This is a headache that must be planned for whenever anyone considers changing the addresses of their Web pages, for whatever reason.

Barry Schwartz at Search Engine Roundtable last week highlighted a discussion at Google Groups in which Google's Adam Lasnik claimed that it only takes them a "couple of weeks for things to smooth out."

Unfortunately, although Google seems to be able to index pages within a few weeks, the past rankings for those pages are not updated in any time close to that, especially for competitive terms, without some additional effort. The primary method to speed this process seems to be gaining links from authority sites to the new URL, in a rapid fashion. Funny because that also seems to be the way out of what some call the fictional Google Sandbox.

The "trick" often used in order to try to lessen the severity of the old pages' loss of rankings is to employ a 302 redirect instead. This causes Google to keep the listing of the previous page within its index, and often in the same position within the rankings.

Some SEO's recommend using this tactic while "building up value" of the new pages through the form of new links that lead to the new URL. Although I have personally seen this work, I would recommend using 301s right away for all pages. It is nice to do this "off-season" if you are a seasonal kind of business, but unfortunately that isn't the case for every one.

I have always hoped to find a larger sample of case studies which show that Google can perform faster than what people consistently forecast as 3 months before original rankings are regained, if ever. Unfortunately, according to our engineers, the clients we have worked with have rarely seen this rapidity in bounce-back-ability. I asked Barry who agreed that 3 months was much closer to the norm.

So will Google allow users of the Webmaster Central portal to maybe jump line when it comes to regaining lost rankings due to URL rewrite or move? Will its algorithm ever speed this process out or will it stay like this to help avoid accidentally ranking pages which have considerable content changes.

Either way, moving to new URLs is something that will cause headaches no matter what, it seems. Everyone involved with the Web site should know that before the redirects are implemented, and other means of driving traffic to the pages should be considered as a top gap. These means include, but are not limited to the use of Paid Search, traditional marketing, as well as banner placement on well-trafficked sites.

Posted by Chris Boggs at 12:14 PM | Permalink

April 11, 2006

DNS Cache Poisoning & Hijacking Search Results

I reported earlier this morning that some scam artists are using DNS Cache Poisoning to redirect your PPC and organic listings to other pages. First reports of this come from WebmasterWorld, where it has been documented that an advertiser was targeted and exploited. Basically, the attacker will poison your DNS cache (if it is possible) and then "redirect a popular search engine to a pop-up ad site," or something similar.

SEO Consultants has a very detailed write up on this vulnerability. You should immediately check your domain names at DNS Reports to ensure you pass for "Open DNS servers" test.

Posted by Barry Schwartz at 8:55 AM | Permalink

DNS Cache Poisoning & Hijacking Search Results

I reported earlier this morning that some scam artists are using DNS Cache Poisoning to redirect your PPC and organic listings to other pages. First reports of this come from WebmasterWorld, where it has been documented that an advertiser was targeted and exploited. Basically, the attacker will poison your DNS cache (if it is possible) and then "redirect a popular search engine to a pop-up ad site," or something similar.

SEO Consultants has a very detailed write up on this vulnerability. You should immediately check your domain names at DNS Reports to ensure you pass for "Open DNS servers" test.

Posted by Kevin Heisler at 8:55 AM | Permalink

DNS Cache Poisoning & Hijacking Search Results

I reported earlier this morning that some scam artists are using DNS Cache Poisoning to redirect your PPC and organic listings to other pages. First reports of this come from WebmasterWorld, where it has been documented that an advertiser was targeted and exploited. Basically, the attacker will poison your DNS cache (if it is possible) and then "redirect a popular search engine to a pop-up ad site," or something similar.

SEO Consultants has a very detailed write up on this vulnerability. You should immediately check your domain names at DNS Reports to ensure you pass for "Open DNS servers" test.

Posted by Kevin Heisler at 8:55 AM | Permalink

DNS Cache Poisoning & Hijacking Search Results

I reported earlier this morning that some scam artists are using DNS Cache Poisoning to redirect your PPC and organic listings to other pages. First reports of this come from WebmasterWorld, where it has been documented that an advertiser was targeted and exploited. Basically, the attacker will poison your DNS cache (if it is possible) and then "redirect a popular search engine to a pop-up ad site," or something similar.

SEO Consultants has a very detailed write up on this vulnerability. You should immediately check your domain names at DNS Reports to ensure you pass for "Open DNS servers" test.

Posted by Kevin Heisler at 8:55 AM | Permalink

November 23, 2005

Getting Search Engine Love From Affiliate Links

A long time issue with those who ponder using affiliates is whether paying for affiliate links will rob them of link juice with the search engines. If the affiliate links make use of tracking codes or redirection, will search engines count these as much as "real" links?

SEO Friendly Affiliate Systems from Greg Boser is a great tutorial on how affiliate links work in various flavors and how to get the ones that will help you with search engines (short answer, use 301 redirection), assuming your affiliates buy into the idea. As Greg explains, giving you the link juice means they might be less likely to show up themselves.

Want to comment or discuss? Visit our Search Engine Watch Forum thread, Affiliate URLs - 301s vs 302s.

Posted by Danny Sullivan at 10:08 AM | Permalink

Getting Search Engine Love From Affiliate Links

A long time issue with those who ponder using affiliates is whether paying for affiliate links will rob them of link juice with the search engines. If the affiliate links make use of tracking codes or redirection, will search engines count these as much as "real" links?

SEO Friendly Affiliate Systems from Greg Boser is a great tutorial on how affiliate links work in various flavors and how to get the ones that will help you with search engines (short answer, use 301 redirection), assuming your affiliates buy into the idea. As Greg explains, giving you the link juice means they might be less likely to show up themselves.

Want to comment or discuss? Visit our Search Engine Watch Forum thread, Affiliate URLs - 301s vs 302s.

Posted by Kevin Heisler at 10:08 AM | Permalink

Getting Search Engine Love From Affiliate Links

A long time issue with those who ponder using affiliates is whether paying for affiliate links will rob them of link juice with the search engines. If the affiliate links make use of tracking codes or redirection, will search engines count these as much as "real" links?

SEO Friendly Affiliate Systems from Greg Boser is a great tutorial on how affiliate links work in various flavors and how to get the ones that will help you with search engines (short answer, use 301 redirection), assuming your affiliates buy into the idea. As Greg explains, giving you the link juice means they might be less likely to show up themselves.

Want to comment or discuss? Visit our Search Engine Watch Forum thread, Affiliate URLs - 301s vs 302s.

Posted by Kevin Heisler at 10:08 AM | Permalink

Getting Search Engine Love From Affiliate Links

A long time issue with those who ponder using affiliates is whether paying for affiliate links will rob them of link juice with the search engines. If the affiliate links make use of tracking codes or redirection, will search engines count these as much as "real" links?

SEO Friendly Affiliate Systems from Greg Boser is a great tutorial on how affiliate links work in various flavors and how to get the ones that will help you with search engines (short answer, use 301 redirection), assuming your affiliates buy into the idea. As Greg explains, giving you the link juice means they might be less likely to show up themselves.

Want to comment or discuss? Visit our Search Engine Watch Forum thread, Affiliate URLs - 301s vs 302s.

Posted by Kevin Heisler at 10:08 AM | Permalink

October 21, 2005

The MSN PageRank 2 Controversy & Search Engines Needing To Offer Domain Management Tools

As part of the current Google update underway, it's been noticed that MSN now has a PageRank score of 2. What's the deal, Google -- decide to pull a little love away from MSN? Not so, says Google's Matt Cutts -- they're actually a PR8. Then why do you see a PR2 score when you go to MSN? Let's break it down, as well as revisit the oft-desired need for search engines to allow site owners to tell them directly which domains should be treated as the same.

Visit MSN at http://www.msn.com with the Google Toolbar installed, and you'll see a PR2 score reported.
MSN.com down to pagerank 2! at WebmasterWorld has some of the discussion this sparked, where the anonymous GoogleGuy from Google puts the blame on redirection that MSN is doing. Look at http://msn.com, and you'll see a PR8 score is reported, we're told.
OK, but if you try that, you get redirected to http://www.msn.com, where it's still PR2.
Answer? You need to get the Google PageRank score for msn.com in another way than trying to reach that page, since the redirection will send you to what's technically a different page, the home page of www.msn.com
How? Google's Matt Cutts, posting over Threadwatch and sounding pretty in sync with GoogleGuy, explains that msn.com is a PR8 site and points to the Future PageRank checker at SEO Tools as a way to see this. (At this point, you're asking "Isn't Matt Cutts GoogleGuy?" For the record, Matt's never publicly laid claim to being GoogleGuy. But since Matt's more active on commenting with things these days, I think it's well time that GoogleGuy step forward with a real name, so that if they are one and the same, there's isn't confusion that two different people are talking. Honestly, at some point we'll have someone citing GoogleGuy, then someone citing Matt against GoogleGuy, which is absurd if they are the same. I and many others do know the real identity of GoogleGuy. I think it's well time everyone knows and hope GoogleGuy will step forward).
Run a test for msn.com using the checker, and you'll get a list of the PageRank scores reported by various Google datacenters, including the server that feeds the toolbar display. All of them are PR8.
Again, you can't see these scores showing up in your browser when trying to go to msn.com, because you get redirected to www.msn.com, which has a different PR score.

All this brings us back to the issue of redirection. MSN is doing a 302 temporary redirect from msn.com to www.msn.com, which can confuse search engines into knowing if they should be treated at the same site. A 301 permanent redirect would be preferred.

But more preferred than that, life would be a lot easier if site owners could simply register all the various domains that may resolve to their "main" domain with Google and other search engines, rather than them having to guess.

People have been wanting this for ages. C'mon Google and Yahoo! You've both made moves to let us submit sitemaps and URLs to be crawled. Let's get with it so we can register domains with you and how they should be treated through some type of program. It so long overdue. That's especially so given that after the last indexing summit, as I've written, the search engines were unable to unify on any common treatment of dealing with redirects.

Posted by Danny Sullivan at 9:21 AM | Permalink

The MSN PageRank 2 Controversy & Search Engines Needing To Offer Domain Management Tools

As part of the current Google update underway, it's been noticed that MSN now has a PageRank score of 2. What's the deal, Google -- decide to pull a little love away from MSN? Not so, says Google's Matt Cutts -- they're actually a PR8. Then why do you see a PR2 score when you go to MSN? Let's break it down, as well as revisit the oft-desired need for search engines to allow site owners to tell them directly which domains should be treated as the same.

Visit MSN at http://www.msn.com with the Google Toolbar installed, and you'll see a PR2 score reported.
MSN.com down to pagerank 2! at WebmasterWorld has some of the discussion this sparked, where the anonymous GoogleGuy from Google puts the blame on redirection that MSN is doing. Look at http://msn.com, and you'll see a PR8 score is reported, we're told.
OK, but if you try that, you get redirected to http://www.msn.com, where it's still PR2.
Answer? You need to get the Google PageRank score for msn.com in another way than trying to reach that page, since the redirection will send you to what's technically a different page, the home page of www.msn.com
How? Google's Matt Cutts, posting over Threadwatch and sounding pretty in sync with GoogleGuy, explains that msn.com is a PR8 site and points to the Future PageRank checker at SEO Tools as a way to see this. (At this point, you're asking "Isn't Matt Cutts GoogleGuy?" For the record, Matt's never publicly laid claim to being GoogleGuy. But since Matt's more active on commenting with things these days, I think it's well time that GoogleGuy step forward with a real name, so that if they are one and the same, there's isn't confusion that two different people are talking. Honestly, at some point we'll have someone citing GoogleGuy, then someone citing Matt against GoogleGuy, which is absurd if they are the same. I and many others do know the real identity of GoogleGuy. I think it's well time everyone knows and hope GoogleGuy will step forward).
Run a test for msn.com using the checker, and you'll get a list of the PageRank scores reported by various Google datacenters, including the server that feeds the toolbar display. All of them are PR8.
Again, you can't see these scores showing up in your browser when trying to go to msn.com, because you get redirected to www.msn.com, which has a different PR score.

All this brings us back to the issue of redirection. MSN is doing a 302 temporary redirect from msn.com to www.msn.com, which can confuse search engines into knowing if they should be treated at the same site. A 301 permanent redirect would be preferred.

But more preferred than that, life would be a lot easier if site owners could simply register all the various domains that may resolve to their "main" domain with Google and other search engines, rather than them having to guess.

People have been wanting this for ages. C'mon Google and Yahoo! You've both made moves to let us submit sitemaps and URLs to be crawled. Let's get with it so we can register domains with you and how they should be treated through some type of program. It so long overdue. That's especially so given that after the last indexing summit, as I've written, the search engines were unable to unify on any common treatment of dealing with redirects.

Posted by Kevin Heisler at 9:21 AM | Permalink

The MSN PageRank 2 Controversy & Search Engines Needing To Offer Domain Management Tools

As part of the current Google update underway, it's been noticed that MSN now has a PageRank score of 2. What's the deal, Google -- decide to pull a little love away from MSN? Not so, says Google's Matt Cutts -- they're actually a PR8. Then why do you see a PR2 score when you go to MSN? Let's break it down, as well as revisit the oft-desired need for search engines to allow site owners to tell them directly which domains should be treated as the same.

Visit MSN at http://www.msn.com with the Google Toolbar installed, and you'll see a PR2 score reported.
MSN.com down to pagerank 2! at WebmasterWorld has some of the discussion this sparked, where the anonymous GoogleGuy from Google puts the blame on redirection that MSN is doing. Look at http://msn.com, and you'll see a PR8 score is reported, we're told.
OK, but if you try that, you get redirected to http://www.msn.com, where it's still PR2.
Answer? You need to get the Google PageRank score for msn.com in another way than trying to reach that page, since the redirection will send you to what's technically a different page, the home page of www.msn.com
How? Google's Matt Cutts, posting over Threadwatch and sounding pretty in sync with GoogleGuy, explains that msn.com is a PR8 site and points to the Future PageRank checker at SEO Tools as a way to see this. (At this point, you're asking "Isn't Matt Cutts GoogleGuy?" For the record, Matt's never publicly laid claim to being GoogleGuy. But since Matt's more active on commenting with things these days, I think it's well time that GoogleGuy step forward with a real name, so that if they are one and the same, there's isn't confusion that two different people are talking. Honestly, at some point we'll have someone citing GoogleGuy, then someone citing Matt against GoogleGuy, which is absurd if they are the same. I and many others do know the real identity of GoogleGuy. I think it's well time everyone knows and hope GoogleGuy will step forward).
Run a test for msn.com using the checker, and you'll get a list of the PageRank scores reported by various Google datacenters, including the server that feeds the toolbar display. All of them are PR8.
Again, you can't see these scores showing up in your browser when trying to go to msn.com, because you get redirected to www.msn.com, which has a different PR score.

All this brings us back to the issue of redirection. MSN is doing a 302 temporary redirect from msn.com to www.msn.com, which can confuse search engines into knowing if they should be treated at the same site. A 301 permanent redirect would be preferred.

But more preferred than that, life would be a lot easier if site owners could simply register all the various domains that may resolve to their "main" domain with Google and other search engines, rather than them having to guess.

People have been wanting this for ages. C'mon Google and Yahoo! You've both made moves to let us submit sitemaps and URLs to be crawled. Let's get with it so we can register domains with you and how they should be treated through some type of program. It so long overdue. That's especially so given that after the last indexing summit, as I've written, the search engines were unable to unify on any common treatment of dealing with redirects.

Posted by Kevin Heisler at 9:21 AM | Permalink

The MSN PageRank 2 Controversy & Search Engines Needing To Offer Domain Management Tools

As part of the current Google update underway, it's been noticed that MSN now has a PageRank score of 2. What's the deal, Google -- decide to pull a little love away from MSN? Not so, says Google's Matt Cutts -- they're actually a PR8. Then why do you see a PR2 score when you go to MSN? Let's break it down, as well as revisit the oft-desired need for search engines to allow site owners to tell them directly which domains should be treated as the same.

Visit MSN at http://www.msn.com with the Google Toolbar installed, and you'll see a PR2 score reported.
MSN.com down to pagerank 2! at WebmasterWorld has some of the discussion this sparked, where the anonymous GoogleGuy from Google puts the blame on redirection that MSN is doing. Look at http://msn.com, and you'll see a PR8 score is reported, we're told.
OK, but if you try that, you get redirected to http://www.msn.com, where it's still PR2.
Answer? You need to get the Google PageRank score for msn.com in another way than trying to reach that page, since the redirection will send you to what's technically a different page, the home page of www.msn.com
How? Google's Matt Cutts, posting over Threadwatch and sounding pretty in sync with GoogleGuy, explains that msn.com is a PR8 site and points to the Future PageRank checker at SEO Tools as a way to see this. (At this point, you're asking "Isn't Matt Cutts GoogleGuy?" For the record, Matt's never publicly laid claim to being GoogleGuy. But since Matt's more active on commenting with things these days, I think it's well time that GoogleGuy step forward with a real name, so that if they are one and the same, there's isn't confusion that two different people are talking. Honestly, at some point we'll have someone citing GoogleGuy, then someone citing Matt against GoogleGuy, which is absurd if they are the same. I and many others do know the real identity of GoogleGuy. I think it's well time everyone knows and hope GoogleGuy will step forward).
Run a test for msn.com using the checker, and you'll get a list of the PageRank scores reported by various Google datacenters, including the server that feeds the toolbar display. All of them are PR8.
Again, you can't see these scores showing up in your browser when trying to go to msn.com, because you get redirected to www.msn.com, which has a different PR score.

All this brings us back to the issue of redirection. MSN is doing a 302 temporary redirect from msn.com to www.msn.com, which can confuse search engines into knowing if they should be treated at the same site. A 301 permanent redirect would be preferred.

But more preferred than that, life would be a lot easier if site owners could simply register all the various domains that may resolve to their "main" domain with Google and other search engines, rather than them having to guess.

People have been wanting this for ages. C'mon Google and Yahoo! You've both made moves to let us submit sitemaps and URLs to be crawled. Let's get with it so we can register domains with you and how they should be treated through some type of program. It so long overdue. That's especially so given that after the last indexing summit, as I've written, the search engines were unable to unify on any common treatment of dealing with redirects.

Posted by Kevin Heisler at 9:21 AM | Permalink

October 17, 2005

Matt Cutts Gives Tips On Moving Sites & Keeping Google Happy

Google's Matt Cutts has moved his blog to a new host, and he shares some advice on how to keep Google happy if you have to do a similar move here. The last step is most important. Once you are sure Googlebots are visiting the new site and not the old, it's safe to shut things down. He also touches on what do to if you need to change domains (short answer, 301 redirect from old to new). By the way, Retaining Traffic after a Web Site Redesign from SearchDay last year has some related tips you may find useful.

Posted by Danny Sullivan at 7:27 PM | Permalink

Matt Cutts Gives Tips On Moving Sites & Keeping Google Happy

Google's Matt Cutts has moved his blog to a new host, and he shares some advice on how to keep Google happy if you have to do a similar move here. The last step is most important. Once you are sure Googlebots are visiting the new site and not the old, it's safe to shut things down. He also touches on what do to if you need to change domains (short answer, 301 redirect from old to new). By the way, Retaining Traffic after a Web Site Redesign from SearchDay last year has some related tips you may find useful.

Posted by Kevin Heisler at 7:27 PM | Permalink

Matt Cutts Gives Tips On Moving Sites & Keeping Google Happy

Google's Matt Cutts has moved his blog to a new host, and he shares some advice on how to keep Google happy if you have to do a similar move here. The last step is most important. Once you are sure Googlebots are visiting the new site and not the old, it's safe to shut things down. He also touches on what do to if you need to change domains (short answer, 301 redirect from old to new). By the way, Retaining Traffic after a Web Site Redesign from SearchDay last year has some related tips you may find useful.

Posted by Kevin Heisler at 7:27 PM | Permalink

Matt Cutts Gives Tips On Moving Sites & Keeping Google Happy

Google's Matt Cutts has moved his blog to a new host, and he shares some advice on how to keep Google happy if you have to do a similar move here. The last step is most important. Once you are sure Googlebots are visiting the new site and not the old, it's safe to shut things down. He also touches on what do to if you need to change domains (short answer, 301 redirect from old to new). By the way, Retaining Traffic after a Web Site Redesign from SearchDay last year has some related tips you may find useful.

Posted by Kevin Heisler at 7:27 PM | Permalink

September 29, 2005

Listings Hijacked At MSN, With A Little Help From Google

Google 302 and MSN from Dave Naylor is chock full of badness on the parts of both Google and MSN, showing how Google redirections are causing it to hijack listings in MSN's search results. Dave gives you the short rundown. Here's the spelled out version, and thanks for his help in assembling it.

Look at this search result at MSN UK for batman animated bean bag.
See how the first result is for this page at Kids UK?
Now look at the URL MSN UK lists for that page: http://groups.google.co.uk/froogle_url?q= http://www.kidsuk.co.uk/shop/catalog/ Batman-Bedding-p-1-c-1288.html %3Fsource%3Dfroogle&fr=AJrr2tQq23-_SJjef Mma5wwNUyhA6FBUGEdlEBymj9jJAAAAAAAAAAA
See the bold part? That shows that MSN believes this page is hosted at groups.google.co.uk.
What's happening is over at Froogle UK, all links you click on there are redirected out of Google and to the destination sites, but...
Google is using 302 temporary redirection, which is causing MSN to let it "hijack" these listings.
To be clear, MSN is NOT listing a Google page, even though the page has a Google URL. Look at the cached copy of that page, and you can see that it is the same page as at Kids UK. But Google has control over the URL in MSN's results.
In other words, should Google lose its mind, it could at any point send MSN a cloaked version of the Kids UK page and likely maintain the ranking while showing human visitors something else entirely. Kids UK is not in control of that listing on MSN, even though it currently leads to the Kids UK site. It's been hijacked by Google! If Google were using a 301 redirection, however, this shouldn't be happening.
Side point. If this is a Froogle UK thing, why does that URL say GROUPS.google.co.uk? Google UK has some domain madness going on. Visit the home page. Click the Froogle link to get this page. Now click the Groups link to reach this page. Notice now how even though you are in Google Groups, the the froogle.co.uk domain is what shows in your address bar. That shouldn't be happening. Other mix-ups like this are leading to the confusion.
Hey! What's MSN doing crawling Froogle anyway? The robots.txt file there should be keeping it out, right? Sure. But if some site has made copies of Froogle results, scraped the content as fodder for a fake blog or something else to attract traffic, MSN might crawl that and thus see the Froogle redirections.

Overall, a nice demonstration of why MSN needs to consider how it handles redirection. My Revisiting Hijacking & Redirects: Moving To A Solution story gives you more background on the hijacking situation as it especially has impacted Google.

I also wrote that story as a lead in for our Indexing Summit 2 session as SES San Jose that was held last month, to see if we could get a standard solution to handing redirection and eliminate these type of problems. I was planning to finally write up what happened at that session next week, and I still will, promise. But here's the summary:

Yahoo: We have a solution (as described in my article) that seems to work.
Google: Matt Cutts wants to use the Yahoo solution but the engineer overseeing how redirections are handled says they've solved it another way. Matt said if you still see it happening, report it to Google, and then he's got some ammunition to say "I told you so" and get the Yahoo solution going. It's been reported at least once already. Bacon polenta on Matt's blog explains that and more important, gives updated instructions on how to report a hijacking in Google's listings.
Ask Jeeves: Thinks it has a handle on the situation and doesn't need to follow the Yahoo solution.
MSN: Didn't take part in the summit.

Want to discuss or comment? Visit our forum thread, Google Hijacks Batman Room Decor Listing At MSN!

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Posted by Danny Sullivan at 1:42 PM | Permalink

Listings Hijacked At MSN, With A Little Help From Google

Google 302 and MSN from Dave Naylor is chock full of badness on the parts of both Google and MSN, showing how Google redirections are causing it to hijack listings in MSN's search results. Dave gives you the short rundown. Here's the spelled out version, and thanks for his help in assembling it.

Look at this search result at MSN UK for batman animated bean bag.
See how the first result is for this page at Kids UK?
Now look at the URL MSN UK lists for that page: http://groups.google.co.uk/froogle_url?q= http://www.kidsuk.co.uk/shop/catalog/ Batman-Bedding-p-1-c-1288.html %3Fsource%3Dfroogle&fr=AJrr2tQq23-_SJjef Mma5wwNUyhA6FBUGEdlEBymj9jJAAAAAAAAAAA
See the bold part? That shows that MSN believes this page is hosted at groups.google.co.uk.
What's happening is over at Froogle UK, all links you click on there are redirected out of Google and to the destination sites, but...
Google is using 302 temporary redirection, which is causing MSN to let it "hijack" these listings.
To be clear, MSN is NOT listing a Google page, even though the page has a Google URL. Look at the cached copy of that page, and you can see that it is the same page as at Kids UK. But Google has control over the URL in MSN's results.
In other words, should Google lose its mind, it could at any point send MSN a cloaked version of the Kids UK page and likely maintain the ranking while showing human visitors something else entirely. Kids UK is not in control of that listing on MSN, even though it currently leads to the Kids UK site. It's been hijacked by Google! If Google were using a 301 redirection, however, this shouldn't be happening.
Side point. If this is a Froogle UK thing, why does that URL say GROUPS.google.co.uk? Google UK has some domain madness going on. Visit the home page. Click the Froogle link to get this page. Now click the Groups link to reach this page. Notice now how even though you are in Google Groups, the the froogle.co.uk domain is what shows in your address bar. That shouldn't be happening. Other mix-ups like this are leading to the confusion.
Hey! What's MSN doing crawling Froogle anyway? The robots.txt file there should be keeping it out, right? Sure. But if some site has made copies of Froogle results, scraped the content as fodder for a fake blog or something else to attract traffic, MSN might crawl that and thus see the Froogle redirections.

Overall, a nice demonstration of why MSN needs to consider how it handles redirection. My Revisiting Hijacking & Redirects: Moving To A Solution story gives you more background on the hijacking situation as it especially has impacted Google.

I also wrote that story as a lead in for our Indexing Summit 2 session as SES San Jose that was held last month, to see if we could get a standard solution to handing redirection and eliminate these type of problems. I was planning to finally write up what happened at that session next week, and I still will, promise. But here's the summary:

Yahoo: We have a solution (as described in my article) that seems to work.
Google: Matt Cutts wants to use the Yahoo solution but the engineer overseeing how redirections are handled says they've solved it another way. Matt said if you still see it happening, report it to Google, and then he's got some ammunition to say "I told you so" and get the Yahoo solution going. It's been reported at least once already. Bacon polenta on Matt's blog explains that and more important, gives updated instructions on how to report a hijacking in Google's listings.
Ask Jeeves: Thinks it has a handle on the situation and doesn't need to follow the Yahoo solution.
MSN: Didn't take part in the summit.

Want to discuss or comment? Visit our forum thread, Google Hijacks Batman Room Decor Listing At MSN!

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Posted by Kevin Heisler at 1:42 PM | Permalink

Listings Hijacked At MSN, With A Little Help From Google

Google 302 and MSN from Dave Naylor is chock full of badness on the parts of both Google and MSN, showing how Google redirections are causing it to hijack listings in MSN's search results. Dave gives you the short rundown. Here's the spelled out version, and thanks for his help in assembling it.

Look at this search result at MSN UK for batman animated bean bag.
See how the first result is for this page at Kids UK?
Now look at the URL MSN UK lists for that page: http://groups.google.co.uk/froogle_url?q= http://www.kidsuk.co.uk/shop/catalog/ Batman-Bedding-p-1-c-1288.html %3Fsource%3Dfroogle&fr=AJrr2tQq23-_SJjef Mma5wwNUyhA6FBUGEdlEBymj9jJAAAAAAAAAAA
See the bold part? That shows that MSN believes this page is hosted at groups.google.co.uk.
What's happening is over at Froogle UK, all links you click on there are redirected out of Google and to the destination sites, but...
Google is using 302 temporary redirection, which is causing MSN to let it "hijack" these listings.
To be clear, MSN is NOT listing a Google page, even though the page has a Google URL. Look at the cached copy of that page, and you can see that it is the same page as at Kids UK. But Google has control over the URL in MSN's results.
In other words, should Google lose its mind, it could at any point send MSN a cloaked version of the Kids UK page and likely maintain the ranking while showing human visitors something else entirely. Kids UK is not in control of that listing on MSN, even though it currently leads to the Kids UK site. It's been hijacked by Google! If Google were using a 301 redirection, however, this shouldn't be happening.
Side point. If this is a Froogle UK thing, why does that URL say GROUPS.google.co.uk? Google UK has some domain madness going on. Visit the home page. Click the Froogle link to get this page. Now click the Groups link to reach this page. Notice now how even though you are in Google Groups, the the froogle.co.uk domain is what shows in your address bar. That shouldn't be happening. Other mix-ups like this are leading to the confusion.
Hey! What's MSN doing crawling Froogle anyway? The robots.txt file there should be keeping it out, right? Sure. But if some site has made copies of Froogle results, scraped the content as fodder for a fake blog or something else to attract traffic, MSN might crawl that and thus see the Froogle redirections.

Overall, a nice demonstration of why MSN needs to consider how it handles redirection. My Revisiting Hijacking & Redirects: Moving To A Solution story gives you more background on the hijacking situation as it especially has impacted Google.

I also wrote that story as a lead in for our Indexing Summit 2 session as SES San Jose that was held last month, to see if we could get a standard solution to handing redirection and eliminate these type of problems. I was planning to finally write up what happened at that session next week, and I still will, promise. But here's the summary:

Yahoo: We have a solution (as described in my article) that seems to work.
Google: Matt Cutts wants to use the Yahoo solution but the engineer overseeing how redirections are handled says they've solved it another way. Matt said if you still see it happening, report it to Google, and then he's got some ammunition to say "I told you so" and get the Yahoo solution going. It's been reported at least once already. Bacon polenta on Matt's blog explains that and more important, gives updated instructions on how to report a hijacking in Google's listings.
Ask Jeeves: Thinks it has a handle on the situation and doesn't need to follow the Yahoo solution.
MSN: Didn't take part in the summit.

Want to discuss or comment? Visit our forum thread, Google Hijacks Batman Room Decor Listing At MSN!

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Posted by Kevin Heisler at 1:42 PM | Permalink

Listings Hijacked At MSN, With A Little Help From Google

Google 302 and MSN from Dave Naylor is chock full of badness on the parts of both Google and MSN, showing how Google redirections are causing it to hijack listings in MSN's search results. Dave gives you the short rundown. Here's the spelled out version, and thanks for his help in assembling it.

Look at this search result at MSN UK for batman animated bean bag.
See how the first result is for this page at Kids UK?
Now look at the URL MSN UK lists for that page: http://groups.google.co.uk/froogle_url?q= http://www.kidsuk.co.uk/shop/catalog/ Batman-Bedding-p-1-c-1288.html %3Fsource%3Dfroogle&fr=AJrr2tQq23-_SJjef Mma5wwNUyhA6FBUGEdlEBymj9jJAAAAAAAAAAA
See the bold part? That shows that MSN believes this page is hosted at groups.google.co.uk.
What's happening is over at Froogle UK, all links you click on there are redirected out of Google and to the destination sites, but...
Google is using 302 temporary redirection, which is causing MSN to let it "hijack" these listings.
To be clear, MSN is NOT listing a Google page, even though the page has a Google URL. Look at the cached copy of that page, and you can see that it is the same page as at Kids UK. But Google has control over the URL in MSN's results.
In other words, should Google lose its mind, it could at any point send MSN a cloaked version of the Kids UK page and likely maintain the ranking while showing human visitors something else entirely. Kids UK is not in control of that listing on MSN, even though it currently leads to the Kids UK site. It's been hijacked by Google! If Google were using a 301 redirection, however, this shouldn't be happening.
Side point. If this is a Froogle UK thing, why does that URL say GROUPS.google.co.uk? Google UK has some domain madness going on. Visit the home page. Click the Froogle link to get this page. Now click the Groups link to reach this page. Notice now how even though you are in Google Groups, the the froogle.co.uk domain is what shows in your address bar. That shouldn't be happening. Other mix-ups like this are leading to the confusion.
Hey! What's MSN doing crawling Froogle anyway? The robots.txt file there should be keeping it out, right? Sure. But if some site has made copies of Froogle results, scraped the content as fodder for a fake blog or something else to attract traffic, MSN might crawl that and thus see the Froogle redirections.

Overall, a nice demonstration of why MSN needs to consider how it handles redirection. My Revisiting Hijacking & Redirects: Moving To A Solution story gives you more background on the hijacking situation as it especially has impacted Google.

I also wrote that story as a lead in for our Indexing Summit 2 session as SES San Jose that was held last month, to see if we could get a standard solution to handing redirection and eliminate these type of problems. I was planning to finally write up what happened at that session next week, and I still will, promise. But here's the summary:

Yahoo: We have a solution (as described in my article) that seems to work.
Google: Matt Cutts wants to use the Yahoo solution but the engineer overseeing how redirections are handled says they've solved it another way. Matt said if you still see it happening, report it to Google, and then he's got some ammunition to say "I told you so" and get the Yahoo solution going. It's been reported at least once already. Bacon polenta on Matt's blog explains that and more important, gives updated instructions on how to report a hijacking in Google's listings.
Ask Jeeves: Thinks it has a handle on the situation and doesn't need to follow the Yahoo solution.
MSN: Didn't take part in the summit.

Want to discuss or comment? Visit our forum thread, Google Hijacks Batman Room Decor Listing At MSN!

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Postscript: I was incorrect on the robots.txt banning. The robots.txt file for Google Groups wouldn't have prevented MSN from crawling Froogle results that can be accessed under that domain. More in the forum thread above.

Posted by Kevin Heisler at 1:42 PM | Permalink

September 28, 2005

Yahoo Paid Inclusion Redirection & Hijacking Confusion

Paid Inclusion Making Yahoo Results Seem Hijacked? looks at the confusing situation one of our forum moderators Jeff Martin found when looking at some listings in Yahoo. They redirected through Business.com until winding up at sites that had nothing to do with that B2B search engine. Jeff also describes more here. What's up? I posted my thoughts in the forum thread. To me, it looks like Yahoo is taking a paid inclusion feed from Business.com -- hence why the click redirects through Business.com before hitting the destination sites. In other words, buy some listings in Business.com, and those are distributed also through Yahoo. That's a long-time tactic. LookSmart long did the same.

Posted by Danny Sullivan at 7:33 AM | Permalink

Yahoo Paid Inclusion Redirection & Hijacking Confusion

Paid Inclusion Making Yahoo Results Seem Hijacked? looks at the confusing situation one of our forum moderators Jeff Martin found when looking at some listings in Yahoo. They redirected through Business.com until winding up at sites that had nothing to do with that B2B search engine. Jeff also describes more here. What's up? I posted my thoughts in the forum thread. To me, it looks like Yahoo is taking a paid inclusion feed from Business.com -- hence why the click redirects through Business.com before hitting the destination sites. In other words, buy some listings in Business.com, and those are distributed also through Yahoo. That's a long-time tactic. LookSmart long did the same.

Posted by Kevin Heisler at 7:33 AM | Permalink

Yahoo Paid Inclusion Redirection & Hijacking Confusion

Paid Inclusion Making Yahoo Results Seem Hijacked? looks at the confusing situation one of our forum moderators Jeff Martin found when looking at some listings in Yahoo. They redirected through Business.com until winding up at sites that had nothing to do with that B2B search engine. Jeff also describes more here. What's up? I posted my thoughts in the forum thread. To me, it looks like Yahoo is taking a paid inclusion feed from Business.com -- hence why the click redirects through Business.com before hitting the destination sites. In other words, buy some listings in Business.com, and those are distributed also through Yahoo. That's a long-time tactic. LookSmart long did the same.

Posted by Kevin Heisler at 7:33 AM | Permalink

Yahoo Paid Inclusion Redirection & Hijacking Confusion

Paid Inclusion Making Yahoo Results Seem Hijacked? looks at the confusing situation one of our forum moderators Jeff Martin found when looking at some listings in Yahoo. They redirected through Business.com until winding up at sites that had nothing to do with that B2B search engine. Jeff also describes more here. What's up? I posted my thoughts in the forum thread. To me, it looks like Yahoo is taking a paid inclusion feed from Business.com -- hence why the click redirects through Business.com before hitting the destination sites. In other words, buy some listings in Business.com, and those are distributed also through Yahoo. That's a long-time tactic. LookSmart long did the same.

Posted by Kevin Heisler at 7:33 AM | Permalink

September 7, 2005

Redirection Resources

Trying to understand more about redirection as it applies to seach marketing? Todd over at Stuntdubl lists a few resources you may want to check out.

Posted by Danny Sullivan at 9:05 AM | Permalink

Redirection Resources

Trying to understand more about redirection as it applies to seach marketing? Todd over at Stuntdubl lists a few resources you may want to check out.

Posted by Kevin Heisler at 9:05 AM | Permalink

Redirection Resources

Trying to understand more about redirection as it applies to seach marketing? Todd over at Stuntdubl lists a few resources you may want to check out.

Posted by Kevin Heisler at 9:05 AM | Permalink

Redirection Resources

Trying to understand more about redirection as it applies to seach marketing? Todd over at Stuntdubl lists a few resources you may want to check out.

Posted by Kevin Heisler at 9:05 AM | Permalink

August 1, 2005

Revisiting Hijacking & Redirects: Moving To A Solution

Cast your mind back to May, when I wrote about the Google AdSense page getting "hijacked" by another site for searches on adsense and google adsense. Why hadn't Google yet solved this problem, I asked, especially when it was something that competitor Yahoo seemed to have cracked?

I talked with Google about the issue in more depth a couple of days later. In short, they're aware that the issue needs to be solved, but they're looking for the best way forward and would like your help.

Moreover, it's something we're going to explore at the Indexing Summit 2 at the SES San Jose show next week, to se if there's an across the board solution for all the major search engines. So any advice and suggestions you have on the topic are welcomed. More on how to contribute is below.

Redirection - From Old To New

To understand the issue, let's revisit the basics. What are redirects, and how are they supposed to operate?

It would be great if pages always had the same URL forever and ever, but that's often not the case. People get new domain names, reorganize their web sites or use a new content delivery system. Those are just some of many reasons that URLs can change.

Change your URL, and people -- or search engines -- can't find the page when they look in the old location. Redirection is the solution. With redirection, a site owner can automatically send requests for the page at the old location to the new one. Think of it as mail forwarding or call forwarding for web pages.

Redirection can be done through a meta refresh tag, but I'm not going to get into that here. Just understand that meta tag redirection is a page-based method of pointing from one place to another. In other words, it's something you put on the actual web page.

Instead, I'm going to instead focus on server-side redirection, where your web server does all the work of pointing from one place to another.

When your server does a redirection (or responds to anything), it sends out a little status code about the request it's processed. You know how when you try to reach a web site that's gone, and you sometimes get a "404 Not Found" error? That's a type of status code a server sends, which in turn may trigger the appearance of that "Not Found" page.

With redirection, there are two main codes that go out, corresponding to the type of redirection happening. These are 301 and 302, numbers you've probably heard if you've been reading about redirection and hijacking issues. Let's look further and what they represent and how things are supposed to go.

301 Permanent Redirect

The W3C guidelines for a 301 "permanent redirect" say that this is for use when a page has been permanently moved and you want people to record the new address in place of the old one.

In other words, say you change domains from superdupersite.com to reallycoolhotsite.com. You want people and search engines to know that the new domain should be used in place of the old one. You'd do a 301 permanent redirect like this:

superdupersite.com --- 301 Permanent Redirect ---> reallycoolhotsite.com

Do that, and it's the URL pointed at (that I've highlighted in bold) that should be retained for use in, say, search listings.

302 Temporary Redirect

The W3C guidelines for a 302 "temporary redirect," as it's commonly though inaccurately referred to, say that this is for use when a page has been temporarily moved to a new location and you want people to KEEP the old address rather than use the new one.

In other words, say one day superdupersite.com gets Slashdotted or receives heavy traffic, too much to keep serving up things as normal. You decide to temporarily send people off to a mirror site, mirror.superdupersite.com. You want people and search engines to reach the new location but not record the address as a permanent change. You'd do a 302 temporary redirect like this:

superdupersite.com --- 302 Temporary Redirect ---> mirror.superdupersite.com

Do that, and it's the site doing the pointing whose URL should be retained for use in search listings.

What Yahoo Does

Those are the official "rules," to some degree, on how redirects are supposed to be handled. However, following them has caused problems with the search engines.

In reaction, Yahoo made changes last year in how it handles redirects, as this slide (PDF file) illustrates. Exactly what Yahoo records will differ from what the W3C suggests, depending on the situation. I'll break that down below.

Yahoo: Redirects Between Domain

The first situation is for redirects between two DIFFERENT web sites or domains. Redirects work like this:

301 Permanent Redirect source-domain --> target-domain = target-domain URL kept and used for listings
302 Temporary Redirect source-domain --> target-domain = target-domain URL kept and used for listings

In the examples above, things work exactly the same. If the "source-domain" (the site doing the redirection) redirects in any way to "target-domain" (the site getting the traffic), the target-domain URL is kept.

This solves any hijacking problem. You can't point at someone else and possibly, as with Google (and likely MSN and Ask Jeeves) somehow get your own URL listed rather than the site you're pointing at. Or more important, somone can't redirect to you and manage to get their own URL listed instead of yours.

Here's what Yahoo's Eric Baldeschwieler, director of software development, emailed me about the change:

We decided to handle all cross domain redirects as permanent redirects to remove possibilities for abuse. We were able to avoid the "hijacking" problem. Also, the webmaster community was vocal in its desire to have good permanent redirect support and we have received very positive feedback on this change

Yahoo: Redirects Within A Domain, Root Pages

Things are more complicated at Yahoo when you redirect within your own web site. First the situation with 301 permanent redirects and root pages or "home" pages:

root-page --> deep-page = rootpage kept and used for listings

What's the logic here? Baldeschwieler said:

When a user searches for a domain, we would like to return a domain root page. This motivates our exceptional handling of domain root pages. We put this in place because it reduces the number of user complaints.

Let's use Amazon as an example. Say you go to Yahoo and search for Amazon by name. You want to reach the Amazon home page, in all likelihood. Type in Amazon.com into your browser to see what the home page URL looks like. It should be something similar to this:

http://www.amazon.com/exec/obidos/subst/home/home.html/LOTSOFNUMBERS

Why isn't the address of the Amazon home page just www.amazon.com? Technically, it is. Enter that address, just the domain name, and the "root" page loads up, the default page the server sends out if you don't give it a specific page. But Amazon redirects requests for that page deep within its site and appends a number for tracking purposes.

Now URLs that show in search results are important to users. Studies have shown that they rely on them for making choices. If you want to reach the Amazon home page, it makes a lot of sense to show a nice, short URL rather than the redirection URL that Amazon uses. That's what Yahoo does with this change. Do a search for Amazon, and you'll see the URL shown is:

www.amazon.com

Yahoo does this by technically breaking the rules, but to me, it's a good reason to break them (and hopefully the "rules" will officially change for search engines).

In contrast, Google follows the rules and so the domain it lists for a search on Amazon looks like this (FYI, the tracking numbers aren't shown because as Google can't be cookied, I believe Amazon doesn't generate a unique code for its spider:

www.amazon.com/exec/obidos/subst/home/home.html

You can see the difference. There are some rare occasions when someone might redirect from their home page and want the "deep" page URL to be used. Yahoo admits this and says perhaps other mechanisms will be found to help solve that. But for the most part, I think this "rule breaking" is a good idea.

Yahoo: Redirects Within A Domain, Deep Pages

Now clear your mind of home pages. What if you want to redirect from one "deep" page within a web site to another deep page. Yahoo does this if you do a 301 permanent redirect:

deep-page --> other-deep-page = the page directed to is used for listings

That makes sense (and follows the rules). If you are redirecting from deep pages within a domain, chances are you really do want the "new" address used in listings.

What if you don't want the new address recorded? Easy. Just use a 302 temporary redirect and it follows this logic:

deep-page --> other-deep-page = the redirect is NOT used for listings

As this is happening WITHIN your domain, Yahoo feels it can follow the rules in this case without risk that someone else is trying to "hijack" your listing, as happens with cross-domain redirection.

What Google Does

How does Google handle redirects? With 301 permanent redirects, it uses the URL of whatever is pointed at. There are no potential hijacking problems with this.

With 302 temporary redirects, technically a search engine should keep the URL of the page doing the pointing as explained. If Google actually did this, the hijacking situation would be far worse than it is now. Anyone could point at anyone else and potentially hijack listings. And what if you encounter multiple sites all temporarily redirecting to another site? Which of the "pointing" domains should be kept?

To date, Google has primarily relied on PageRank to help sort the situation out. It generally assumes that the page with the highest PageRank score is the URL that should be used in search listings. Fair to say, this generally works out to be the case.

With Google's much publicized situation in May, there were a number of very unusual glitches that caused its AdSense page to end up having a lower PageRank score than the person pointing at it. Nevertheless, with so many sites out there, such unusual situations can add up.

Solving The Problem

Back to my original question. Why not do what Yahoo does? Perhaps that's what Google will do. Google said when I spoke with it after the May incident that it wanted to spend the next few months getting feedback and experimenting with what seems to be the best solution. Indeed, it has been doing this.

What's so hard? Consider this situation:

Someone registers a domain name -- myname.com
Someone temporarily redirects from that domain to the home page of their blog, such as radio.weblogs.com/30383482/

Ideally, you'd want to keep the actual domain name used for listings, at least for the home page, rather than pointing at that big giant URL. Following the "rules" for a 302 permanent redirect allows this. And the Yahoo solution doesn't work because this is a cross-domain situation.

Dave Naylor, one of our SEW Forum moderators, recently summed it up even better from someone who was redirecting between two domains:

A long long time ago in a distant valley a young search engineer decides that the Google surfers would be better off seeing www.johnnybladeproductions.com rather than seeing johnnybladeproductions.home.att.net, and for a long time everyone was happy.. the webmaster was happy because Google showed his www. and not the free host. Then people realized that pointing a domain with a 302 would replace other people's URLs...and the whole 302 hijack problems started.

The "everyone was happy days" may have changed but not the fact that there can be unusual situations that come up requiring some thought on how to do redirects. Google wants to explore these more before settling on changes.

Ideally, the other search engines would look at them and all come together with a standard. Indeed, while Yahoo has made changes, those could change again. Emailed Yahoo's Baldeschwieler:

Setting redirect handling policy requires balancing the needs of our users and webmasters. The first priority is to answer our users' queries. We also strive to give webmasters as much control as possible of their contents' representation in our index. Hence we are as open, transparent and standards compliant as possible.

I'd like to emphasize that redirect policy handling is not something we have set in stone. We continue to tune our approach as our systems improve and we receive feedback on our current policies. When we last reviewed our policy, we set out a set of principles which have guided our recent redirect handling decisions. Our current implementation choices were driven by the following principles.

Fix the Problems NOW
Don't wait for the perfect solution, work on creating one and remain flexible.
Pay attention to community feedback and user experience, not just standards when designing solutions.
Webmasters and tool vendors often ignore or reinterpret existing standards

Where possible, conform to standards and give control to webmasters. In cases where the above principles don't suggest a divergence from the standard, we conform to it. In all cases, keep web masters informed of how we are handling redirects.

More Background & Your Feedback

For more on redirection issues, I highly recommend Claus Schmidt's Page Hijack: The 302 Exploit, Redirects and Google. He has lots of information there and has played a major role in helping people understand the problem that redirections can cause. You can also see:

Those are blog posts that reference past forum discussions and information from across the web on the topic, which has just continued to grow over the past year and a half.

In this article, I've focused on the situation with Google and Yahoo. I haven't yet have the chance to talk with Ask Jeeves and Microsoft on the topic, but I hoped this would be a good starting point to go forward.

In particular, I hope you can help. Please provide your comments, suggestions and unusual situations you think need to be considered over in this forum thread: Indexing Summit 2: Give Your Feedback On Handling Redirects. Google especially is looking for that feedback. Perhaps a content-location header solution makes sense. Perhaps there are other solutions people have out there.

I'll be mining that thread for possible solutions that we'll explore at Indexing Summit 2, which will be happening at the next Search Engine Strategies show in San Jose this August. Redirection is one of the two key subjects of that. So please speak up now in the thread, so I can bring your voice out along with the others at the actual summit next month.

Posted by Danny Sullivan at 1:03 PM | Permalink

Revisiting Hijacking & Redirects: Moving To A Solution

Cast your mind back to May, when I wrote about the Google AdSense page getting "hijacked" by another site for searches on adsense and google adsense. Why hadn't Google yet solved this problem, I asked, especially when it was something that competitor Yahoo seemed to have cracked?

I talked with Google about the issue in more depth a couple of days later. In short, they're aware that the issue needs to be solved, but they're looking for the best way forward and would like your help.

Moreover, it's something we're going to explore at the Indexing Summit 2 at the SES San Jose show next week, to se if there's an across the board solution for all the major search engines. So any advice and suggestions you have on the topic are welcomed. More on how to contribute is below.

Redirection - From Old To New

To understand the issue, let's revisit the basics. What are redirects, and how are they supposed to operate?

It would be great if pages always had the same URL forever and ever, but that's often not the case. People get new domain names, reorganize their web sites or use a new content delivery system. Those are just some of many reasons that URLs can change.

Change your URL, and people -- or search engines -- can't find the page when they look in the old location. Redirection is the solution. With redirection, a site owner can automatically send requests for the page at the old location to the new one. Think of it as mail forwarding or call forwarding for web pages.

Redirection can be done through a meta refresh tag, but I'm not going to get into that here. Just understand that meta tag redirection is a page-based method of pointing from one place to another. In other words, it's something you put on the actual web page.

Instead, I'm going to instead focus on server-side redirection, where your web server does all the work of pointing from one place to another.

When your server does a redirection (or responds to anything), it sends out a little status code about the request it's processed. You know how when you try to reach a web site that's gone, and you sometimes get a "404 Not Found" error? That's a type of status code a server sends, which in turn may trigger the appearance of that "Not Found" page.

With redirection, there are two main codes that go out, corresponding to the type of redirection happening. These are 301 and 302, numbers you've probably heard if you've been reading about redirection and hijacking issues. Let's look further and what they represent and how things are supposed to go.

301 Permanent Redirect

The W3C guidelines for a 301 "permanent redirect" say that this is for use when a page has been permanently moved and you want people to record the new address in place of the old one.

In other words, say you change domains from superdupersite.com to reallycoolhotsite.com. You want people and search engines to know that the new domain should be used in place of the old one. You'd do a 301 permanent redirect like this:

superdupersite.com --- 301 Permanent Redirect ---> reallycoolhotsite.com

Do that, and it's the URL pointed at (that I've highlighted in bold) that should be retained for use in, say, search listings.

302 Temporary Redirect

The W3C guidelines for a 302 "temporary redirect," as it's commonly though inaccurately referred to, say that this is for use when a page has been temporarily moved to a new location and you want people to KEEP the old address rather than use the new one.

In other words, say one day superdupersite.com gets Slashdotted or receives heavy traffic, too much to keep serving up things as normal. You decide to temporarily send people off to a mirror site, mirror.superdupersite.com. You want people and search engines to reach the new location but not record the address as a permanent change. You'd do a 302 temporary redirect like this:

superdupersite.com --- 302 Temporary Redirect ---> mirror.superdupersite.com

Do that, and it's the site doing the pointing whose URL should be retained for use in search listings.

What Yahoo Does

Those are the official "rules," to some degree, on how redirects are supposed to be handled. However, following them has caused problems with the search engines.

In reaction, Yahoo made changes last year in how it handles redirects, as this slide (PDF file) illustrates. Exactly what Yahoo records will differ from what the W3C suggests, depending on the situation. I'll break that down below.

Yahoo: Redirects Between Domain

The first situation is for redirects between two DIFFERENT web sites or domains. Redirects work like this:

301 Permanent Redirect source-domain --> target-domain = target-domain URL kept and used for listings
302 Temporary Redirect source-domain --> target-domain = target-domain URL kept and used for listings

In the examples above, things work exactly the same. If the "source-domain" (the site doing the redirection) redirects in any way to "target-domain" (the site getting the traffic), the target-domain URL is kept.

This solves any hijacking problem. You can't point at someone else and possibly, as with Google (and likely MSN and Ask Jeeves) somehow get your own URL listed rather than the site you're pointing at. Or more important, somone can't redirect to you and manage to get their own URL listed instead of yours.

Here's what Yahoo's Eric Baldeschwieler, director of software development, emailed me about the change:

We decided to handle all cross domain redirects as permanent redirects to remove possibilities for abuse. We were able to avoid the "hijacking" problem. Also, the webmaster community was vocal in its desire to have good permanent redirect support and we have received very positive feedback on this change

Yahoo: Redirects Within A Domain, Root Pages

Things are more complicated at Yahoo when you redirect within your own web site. First the situation with 301 permanent redirects and root pages or "home" pages:

root-page --> deep-page = rootpage kept and used for listings

What's the logic here? Baldeschwieler said:

When a user searches for a domain, we would like to return a domain root page. This motivates our exceptional handling of domain root pages. We put this in place because it reduces the number of user complaints.

Let's use Amazon as an example. Say you go to Yahoo and search for Amazon by name. You want to reach the Amazon home page, in all likelihood. Type in Amazon.com into your browser to see what the home page URL looks like. It should be something similar to this:

http://www.amazon.com/exec/obidos/subst/home/home.html/LOTSOFNUMBERS

Why isn't the address of the Amazon home page just www.amazon.com? Technically, it is. Enter that address, just the domain name, and the "root" page loads up, the default page the server sends out if you don't give it a specific page. But Amazon redirects requests for that page deep within its site and appends a number for tracking purposes.

Now URLs that show in search results are important to users. Studies have shown that they rely on them for making choices. If you want to reach the Amazon home page, it makes a lot of sense to show a nice, short URL rather than the redirection URL that Amazon uses. That's what Yahoo does with this change. Do a search for Amazon, and you'll see the URL shown is:

www.amazon.com

Yahoo does this by technically breaking the rules, but to me, it's a good reason to break them (and hopefully the "rules" will officially change for search engines).

In contrast, Google follows the rules and so the domain it lists for a search on Amazon looks like this (FYI, the tracking numbers aren't shown because as Google can't be cookied, I believe Amazon doesn't generate a unique code for its spider:

www.amazon.com/exec/obidos/subst/home/home.html

You can see the difference. There are some rare occasions when someone might redirect from their home page and want the "deep" page URL to be used. Yahoo admits this and says perhaps other mechanisms will be found to help solve that. But for the most part, I think this "rule breaking" is a good idea.

Yahoo: Redirects Within A Domain, Deep Pages

Now clear your mind of home pages. What if you want to redirect from one "deep" page within a web site to another deep page. Yahoo does this if you do a 301 permanent redirect:

deep-page --> other-deep-page = the page directed to is used for listings

That makes sense (and follows the rules). If you are redirecting from deep pages within a domain, chances are you really do want the "new" address used in listings.

What if you don't want the new address recorded? Easy. Just use a 302 temporary redirect and it follows this logic:

deep-page --> other-deep-page = the redirect is NOT used for listings

As this is happening WITHIN your domain, Yahoo feels it can follow the rules in this case without risk that someone else is trying to "hijack" your listing, as happens with cross-domain redirection.

What Google Does

How does Google handle redirects? With 301 permanent redirects, it uses the URL of whatever is pointed at. There are no potential hijacking problems with this.

With 302 temporary redirects, technically a search engine should keep the URL of the page doing the pointing as explained. If Google actually did this, the hijacking situation would be far worse than it is now. Anyone could point at anyone else and potentially hijack listings. And what if you encounter multiple sites all temporarily redirecting to another site? Which of the "pointing" domains should be kept?

To date, Google has primarily relied on PageRank to help sort the situation out. It generally assumes that the page with the highest PageRank score is the URL that should be used in search listings. Fair to say, this generally works out to be the case.

With Google's much publicized situation in May, there were a number of very unusual glitches that caused its AdSense page to end up having a lower PageRank score than the person pointing at it. Nevertheless, with so many sites out there, such unusual situations can add up.

Solving The Problem

Back to my original question. Why not do what Yahoo does? Perhaps that's what Google will do. Google said when I spoke with it after the May incident that it wanted to spend the next few months getting feedback and experimenting with what seems to be the best solution. Indeed, it has been doing this.

What's so hard? Consider this situation:

Someone registers a domain name -- myname.com
Someone temporarily redirects from that domain to the home page of their blog, such as radio.weblogs.com/30383482/

Ideally, you'd want to keep the actual domain name used for listings, at least for the home page, rather than pointing at that big giant URL. Following the "rules" for a 302 permanent redirect allows this. And the Yahoo solution doesn't work because this is a cross-domain situation.

Dave Naylor, one of our SEW Forum moderators, recently summed it up even better from someone who was redirecting between two domains:

A long long time ago in a distant valley a young search engineer decides that the Google surfers would be better off seeing www.johnnybladeproductions.com rather than seeing johnnybladeproductions.home.att.net, and for a long time everyone was happy.. the webmaster was happy because Google showed his www. and not the free host. Then people realized that pointing a domain with a 302 would replace other people's URLs...and the whole 302 hijack problems started.

The "everyone was happy days" may have changed but not the fact that there can be unusual situations that come up requiring some thought on how to do redirects. Google wants to explore these more before settling on changes.

Ideally, the other search engines would look at them and all come together with a standard. Indeed, while Yahoo has made changes, those could change again. Emailed Yahoo's Baldeschwieler:

Setting redirect handling policy requires balancing the needs of our users and webmasters. The first priority is to answer our users' queries. We also strive to give webmasters as much control as possible of their contents' representation in our index. Hence we are as open, transparent and standards compliant as possible.

I'd like to emphasize that redirect policy handling is not something we have set in stone. We continue to tune our approach as our systems improve and we receive feedback on our current policies. When we last reviewed our policy, we set out a set of principles which have guided our recent redirect handling decisions. Our current implementation choices were driven by the following principles.

Fix the Problems NOW
Don't wait for the perfect solution, work on creating one and remain flexible.
Pay attention to community feedback and user experience, not just standards when designing solutions.
Webmasters and tool vendors often ignore or reinterpret existing standards

Where possible, conform to standards and give control to webmasters. In cases where the above principles don't suggest a divergence from the standard, we conform to it. In all cases, keep web masters informed of how we are handling redirects.

More Background & Your Feedback

For more on redirection issues, I highly recommend Claus Schmidt's Page Hijack: The 302 Exploit, Redirects and Google. He has lots of information there and has played a major role in helping people understand the problem that redirections can cause. You can also see:

Those are blog posts that reference past forum discussions and information from across the web on the topic, which has just continued to grow over the past year and a half.

In this article, I've focused on the situation with Google and Yahoo. I haven't yet have the chance to talk with Ask Jeeves and Microsoft on the topic, but I hoped this would be a good starting point to go forward.

In particular, I hope you can help. Please provide your comments, suggestions and unusual situations you think need to be considered over in this forum thread: Indexing Summit 2: Give Your Feedback On Handling Redirects. Google especially is looking for that feedback. Perhaps a content-location header solution makes sense. Perhaps there are other solutions people have out there.

I'll be mining that thread for possible solutions that we'll explore at Indexing Summit 2, which will be happening at the next Search Engine Strategies show in San Jose this August. Redirection is one of the two key subjects of that. So please speak up now in the thread, so I can bring your voice out along with the others at the actual summit next month.

Posted by Kevin Heisler at 1:03 PM | Permalink

Revisiting Hijacking & Redirects: Moving To A Solution

Cast your mind back to May, when I wrote about the Google AdSense page getting "hijacked" by another site for searches on adsense and google adsense. Why hadn't Google yet solved this problem, I asked, especially when it was something that competitor Yahoo seemed to have cracked?

I talked with Google about the issue in more depth a couple of days later. In short, they're aware that the issue needs to be solved, but they're looking for the best way forward and would like your help.

Moreover, it's something we're going to explore at the Indexing Summit 2 at the SES San Jose show next week, to se if there's an across the board solution for all the major search engines. So any advice and suggestions you have on the topic are welcomed. More on how to contribute is below.

Redirection - From Old To New

To understand the issue, let's revisit the basics. What are redirects, and how are they supposed to operate?

It would be great if pages always had the same URL forever and ever, but that's often not the case. People get new domain names, reorganize their web sites or use a new content delivery system. Those are just some of many reasons that URLs can change.

Change your URL, and people -- or search engines -- can't find the page when they look in the old location. Redirection is the solution. With redirection, a site owner can automatically send requests for the page at the old location to the new one. Think of it as mail forwarding or call forwarding for web pages.

Redirection can be done through a meta refresh tag, but I'm not going to get into that here. Just understand that meta tag redirection is a page-based method of pointing from one place to another. In other words, it's something you put on the actual web page.

Instead, I'm going to instead focus on server-side redirection, where your web server does all the work of pointing from one place to another.

When your server does a redirection (or responds to anything), it sends out a little status code about the request it's processed. You know how when you try to reach a web site that's gone, and you sometimes get a "404 Not Found" error? That's a type of status code a server sends, which in turn may trigger the appearance of that "Not Found" page.

With redirection, there are two main codes that go out, corresponding to the type of redirection happening. These are 301 and 302, numbers you've probably heard if you've been reading about redirection and hijacking issues. Let's look further and what they represent and how things are supposed to go.

301 Permanent Redirect

The W3C guidelines for a 301 "permanent redirect" say that this is for use when a page has been permanently moved and you want people to record the new address in place of the old one.

In other words, say you change domains from superdupersite.com to reallycoolhotsite.com. You want people and search engines to know that the new domain should be used in place of the old one. You'd do a 301 permanent redirect like this:

superdupersite.com --- 301 Permanent Redirect ---> reallycoolhotsite.com

Do that, and it's the URL pointed at (that I've highlighted in bold) that should be retained for use in, say, search listings.

302 Temporary Redirect

The W3C guidelines for a 302 "temporary redirect," as it's commonly though inaccurately referred to, say that this is for use when a page has been temporarily moved to a new location and you want people to KEEP the old address rather than use the new one.

In other words, say one day superdupersite.com gets Slashdotted or receives heavy traffic, too much to keep serving up things as normal. You decide to temporarily send people off to a mirror site, mirror.superdupersite.com. You want people and search engines to reach the new location but not record the address as a permanent change. You'd do a 302 temporary redirect like this:

superdupersite.com --- 302 Temporary Redirect ---> mirror.superdupersite.com

Do that, and it's the site doing the pointing whose URL should be retained for use in search listings.

What Yahoo Does

Those are the official "rules," to some degree, on how redirects are supposed to be handled. However, following them has caused problems with the search engines.

In reaction, Yahoo made changes last year in how it handles redirects, as this slide (PDF file) illustrates. Exactly what Yahoo records will differ from what the W3C suggests, depending on the situation. I'll break that down below.

Yahoo: Redirects Between Domain

The first situation is for redirects between two DIFFERENT web sites or domains. Redirects work like this:

301 Permanent Redirect source-domain --> target-domain = target-domain URL kept and used for listings
302 Temporary Redirect source-domain --> target-domain = target-domain URL kept and used for listings

In the examples above, things work exactly the same. If the "source-domain" (the site doing the redirection) redirects in any way to "target-domain" (the site getting the traffic), the target-domain URL is kept.

This solves any hijacking problem. You can't point at someone else and possibly, as with Google (and likely MSN and Ask Jeeves) somehow get your own URL listed rather than the site you're pointing at. Or more important, somone can't redirect to you and manage to get their own URL listed instead of yours.

Here's what Yahoo's Eric Baldeschwieler, director of software development, emailed me about the change:

We decided to handle all cross domain redirects as permanent redirects to remove possibilities for abuse. We were able to avoid the "hijacking" problem. Also, the webmaster community was vocal in its desire to have good permanent redirect support and we have received very positive feedback on this change

Yahoo: Redirects Within A Domain, Root Pages

Things are more complicated at Yahoo when you redirect within your own web site. First the situation with 301 permanent redirects and root pages or "home" pages:

root-page --> deep-page = rootpage kept and used for listings

What's the logic here? Baldeschwieler said:

When a user searches for a domain, we would like to return a domain root page. This motivates our exceptional handling of domain root pages. We put this in place because it reduces the number of user complaints.

Let's use Amazon as an example. Say you go to Yahoo and search for Amazon by name. You want to reach the Amazon home page, in all likelihood. Type in Amazon.com into your browser to see what the home page URL looks like. It should be something similar to this:

http://www.amazon.com/exec/obidos/subst/home/home.html/LOTSOFNUMBERS

Why isn't the address of the Amazon home page just www.amazon.com? Technically, it is. Enter that address, just the domain name, and the "root" page