How Search Engines Crawl the websites ?

Question

I am creating a Multilingual web site and I use a resource manager for each language.

when user select a language all pages use the selected resource bondles.

as entire sites only is available in one language,how search engines crawl the other languages ? or does search engine crawl optional provided languages ?

Answer 1

you might be using cookies/sessions for remembering selected language, right? Neither of them affects search engine. They simply ignore cookies. However, If u rely on session variable for remembering selected language, in the absence of cookies each time new session will be created canceling the language selection.

• Ankit

Answer 2

As you know, when you have a static multilingual website that has separate page for each language, you don't have any problem with search engines. Whereas, each page has an unique url.

But in dynamic applications, you don't have separate page for each language and have to use resource instead, you can add a new language or remove an already existing language and so on.

Therefore, we have to use Url Rewriter/Routing for generating unique url for each language. Check the following example out.

Suppose we have a webform in the following url and our application supports two languages (e.g. English United States en-US, English Great Britain en-GB).

www.domain.com/home.aspx

There are some problems, we have permanent url for all of languages. Thus, search engines will be index the default language anyway. The solution is simple, you have to generate separate url for each language by using Url Rewriter/Routing as follows.

www.domain.com/{country}/{language}/home.aspx

Afterwards, you have to inference the specified culture name from the above url and set the current Culture and UICulture properties. Thus, the requested page will be shown in desired language.

The sitemap should be generate programmatically and uses the same way as above, in this case.

www.domain.com/{country}/{language}/sitemap.xml

You have to inference the specified culture from the above url and generate sitemap dependent on culture. To introduce available sitemaps to the search engines you have to use robots.txt that should be generate programmatically as well.