Dynamic Content And Search Engine Optimization

Question

I have a web forum that I have coded in Python for the App Engine platform. I have noticed that it is not being index well by Google and am trying to fix that. I have Google Webmaster to submit a sitemap with almost 400 URLs but apparently only 8 were index! I even get a warning stating:

All the URLs in your Sitemap are marked as having dynamic content (the value of is "always"). Because dynamic content is difficult for search engines to crawl and index, this may impact your site's performance in search results. Check your Sitemap to make sure your site information is correct.

One thing I am considering is the way my URLs are formed. Almost all URLs use arguments and I allocate each post, thread, forum, user etc... an ID. So for example one of my forums is:

http://silicon.appspot.com/readforum?id=2075

Where the forum's ID is 2075. I have heard this is bad practice so am considering changing this but am not sure whether it will make any difference. Could someone give me some hints in relation to how to get Google to index my enire site?

Thanks!

Answer 1

URL content is part of the picture, but only a small part.

Having the URL say readforum2106 instead of /readforum?id=2106 doesn't do anything, unless someone is searching for the word "readforum". To have the URL really help, you need to include some good search terms in the text, which just doesn't make sense for forums.

I don't think URL mapping is really going to help you much. Maybe you should be looking at ways to dynamically generate a site map?

Answer 2

The real thing to look out for it stuff like:

 http://www.example.com/forum.py?thread=1000
 http://www.example.com/forum.py?thread=1000&mode=printer

The web crawler sees two different URLs with the same content and assume you are trying to do something fishy. You need to make sure your "Print this Page" links have rel="nofollow" in them. The goal is to have only one link per unique set of data on your website.

Forums of course are going to be a problem of duplicate content all the time. But for articles/blogs you should be able to approach a 1:1 ratio of pages to URLs.

Answer 3

That statistic in Google Webmaster Tools is never up to date. The following Google search shows that almost all of your sitemap URLs are indexed, and therefore you do not have a problem:

site:silicon.appspot.com