If I search for my companies domain on google, it returns a different domain. It might be our fault. help

Question

Hi - I have recently joined a company, and have got a new task. To work out how googlebot has dropped us.

We have a server, if anyone, googlebot or user, vists the site, it does a geo-location and if the IP is USA it serves domainX.com (US brand) otherwise domainY.com (UK brand). Yes if you give one domain, in New York, you will be 301'd to another domain. There is some good branding reason I am told.

If we enter into google [ domainX.com ] our site is at the top of the rankings. As you would expect if you put your own domain name in.

But if we search for [ domainY.com ] (the UK version) the domainX still appears at the top of the search results. In fact the links to domainY (Uk version) do not appear till page 30 or so.

The big boss does not want a landing page and a 'do you want the US or UK'. I might not have a choice.

But I really do not understand - does googlebot only operate from US IP addresses? DO I need to do something else? IS this flat out a bad idea. Apparently it used to work a couple of months ago.

This is a multi-million company in a safe market, not trying to do funny SEO work (in fact we have a field sales channel - almost all sales come from people wearing out shoe leather)

getting desperate - any pointers welcomed.

Answer 1

How about sitemaps?

There is a specific sitemap protocol, you may want to consult.

Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.

Answer 2

It appears that whenever Gbot goes to your site it is automatically redirected to the US domain. This makes sense since the Gbot is most likely in the US. It's probably having a hard time finding the UK domain since it never visits it. Maybe check for user agent and do something different if the bot is visiting vs a non-bot?

Answer 3

have you tried the search from http://www.google.co.uk/

Answer 4

Anytime the Googlebot follows a link to one of your websites, it get redirected to the US version (assuming Google is operating out of the US). Therefore the US version gets all the PageRank credit for those incoming links, and the UK version gets none.

To solve: check the UserAgent and don't perform a redirect for know robots like Google.

Also, your redirect probably shouldn't be a 301 (permanent). A 302 (temporary) redirect would be more appropriate.

Answer 5

What's very important in my experience is don't move any pages, add new redirects or move sites in a rush. Google tends to be very quickly very displeased with things like that.

Then, you need to remember that Google does Geolocation too, so any search query you make from the US to "see where you are" is automatically "tainted" and likely to show results optimized for the US. As far as I know, this is regardless which Google domain you enter. So you need a UK based IP address or proxy to get realistic UK results.

Google Bot is likely to have a US address, so maybe the setup you have, 301'ing UK customers, won't work for the search engine.

Ideas:

• Play around with "en/us" and "en/gb" (or whatever the exact code is) language strings in page meta tags and "lang" attributes - I have no idea whether that will sort US and UK but maybe worth some research

• Place the UK related content on a server actutally located in the UK - not the management's favourite solution I imagine but maybe worth it.

• Talk management into manual selection of country sites

• Set up sitemaps as already recommended

• Get a Google Webmaster account to get some impression on how Google is seeing your site.

Answer 6

HTTP 301 is a permanent move. As there are plenty of google bots around the world, some will see the US site and some will see the UK site, and it might be confusing for them to experience that some see it permanently moved in one direction and some see it permanently moved in the other direction. Perhaps you should consider any of the temporary redirects HTTP 302, 303 or 307 instead.

Answer 7

This might not be as bad as you think. Don't forget that you are typing in a US and a UK domain name both from within the US; that experiment doesn't mimic practical use though, i wouldn't think. Usually, someone in the UK would key in the DN w/ the '.co.uk' cctld, likewise, someone from the US, the DN w/ the .com tld. Google is pulling those two sets of search results (UK vs. US) from two different 'databases.' I suppose that means that the algorithms that generate those results must be different. So for instance, here in Minsk, i just typed 'google.com' in my browser toolbar window and google redirects me to google.com.by. I can always opt out by clicking on "go to google.com" just below the query window, but in any event, Google's purpose is obviously to serve results to users from the physically nearest data store, and also to localize the results. If i were in the US and I typed 'Adidas.com' in the Google search box, I am certain that adidas.com would be first. When I type 'Adidas.com' in the Google search box as i have done just now (which again has the DN 'google.com.by', even though i typed 'google.com' in my browser's toolbar window, adidas.com is not first, nor even in the top 10. But i just keyed in 'adidas.by' and of course it is first, and the DN is pointing to a Russian language site. So this is good. How does 'adidas.by' do in 'google.com' from the US? I probably don't care as much, and in any event, i might not want to tweak the Site that the .by DN points just to raise it in the US, because i might bring it down here, where the DN (the site it points to actually) matters far more.

Yes a 301 redirect is fairly benign search-engine-wise. Another common technique to deal with this issue is via subdomains, which is simple (everything from a single domain name)--users seems to find it non-intrusive, and search engines seem to favor it among competing techniques. So, e.g., all traffic goes initially to the YourCoSite.com then to /ru, /de, /fr, /by, etc. either via user input ("Choose Your Country") or automatically depending on the user's country identified by IP address, keyboard/language settings, character encoding, etc.)

Answer 8

Thank you for your comments. We have replaced the redirect with a landing page that presents US susers with the option of going to the branded site. Our guess is that a redirect from domainA to domainB tells google domainA no longer is used. We should see some results soon.