Should a highly dynamic website that is constantly generating new pages use a sitemap? If so, how does a site like stackoverflow.com go about regenerating a sitemap? It seems like it would be a drain on precious server resources if it was constantly regenerating a sitemap every time someone adds a question. Does it generate a new sitemap at set intervals (e.g. every four hours)? I'm very curious how large, dynamic websites make this work.
A:
I would only create a site map for the more static pages of the site. For example on StackOverflow a sitemap could showlinks for the FAQ, About, Questions, Tags, Users, etc... pages but not show links to the actual questions, or all the tags, and the various users.
MitMaro
2009-07-08 17:18:23
That misses the purpose of a sitemap.xml file.
ceejayoz
2009-07-08 17:19:19
@MitMaro: I agree with your point (see my similar posting). not sure why you got downvoted; +1 to counter.
McWafflestix
2009-07-08 17:21:38
@ceejayoz: He didn't mention a sitemap.xml file. And even if he did the purpose of a xml sitemap is to allow search engines to index highly dynamic pages or pages with no external links to it. By providing a sitemap to the main sections of a site (for example the questions page on stack overflow) the search engines crawlers can then index the dynamic content on that page (the questions) and hence the sitemap has done what it was supposed to do.
MitMaro
2009-07-08 17:32:41
@MitMaro XML sitemaps do more than that. They also give Google more information on links they already know about. You can, for example, tell Google that a specific link should be reindexed more or less frequently, its last modification date, its importance to the site, etc.
ceejayoz
2009-07-08 17:34:45
I know but that didn't have relevance to my point and these comments are limiting in length.
MitMaro
2009-07-08 18:25:08
A:
Even on something like StackOverflow, there is a certain amount of static organization; there are FAQs, tag pages, question pages, user pages, badge pages, etc.; I'd say in a very dynamic site, the best way to approach a sitemap would be to have a map of the categorizations; each node in the sitemap can point to a page of the dynamically generated data (a node for a question page, a node for a user page, etc.).
Of course, a sitemap may not even be appropriate for a given site; there's a certain amount of judgment call required there.
McWafflestix
2009-07-08 17:20:26
I countered your down vote as well. I guess someone disagrees with us..lol
MitMaro
2009-07-08 17:26:59
@ceejayoz: yup, apparently, however, I think both MitMaro and I answered the question the OP asked; as it turns out, they wanted specificity, but they didn't specify the specificity they wanted, so...
McWafflestix
2009-07-08 18:01:30
@McWafflestix So you're going to leave the downvotes on the answers that correctly understood and answered the original poster's question as he intended it? Way to abuse the system...
ceejayoz
2009-07-08 19:24:30
+2
A:
There's no need to regenerate the Google sitemap XML each time a question is posted. It's far simpler just to have the XML file generated on-demand directly from the database (and a little caching).
To reduce load, the sitemap can be split into many sitemaps. Partitioning it by day/month would allow you to tell Google to retrieve today's sitemap frequently, but only fetch the sitemap from six months ago once in a while.
ceejayoz
2009-07-08 17:20:37
It's implied in the question. No "large, dynamic website" would ever add every question posted to it into a user-accessible sitemap.
ceejayoz
2009-07-08 17:26:02
this is a good answer. i would've accepted it, but Robert's is more nicely formatted with shiny hyperlinks and a quote box!
Andrew Garrison
2009-07-08 17:44:23
+5
A:
On Stackoverflow et al, a sitemap.xml file is created which contains a link to every question posted on the system. When a new question is posted, they simply append another entry to the end of the sitemap file. It isn't that resource intensive to add to the end of the file but the file is quite large.
That is the only way search engines like Google can effectively crawl the site.
Jeff Atwood talks about it in a blog post: The Importance of Sitemaps
This is from Google's webmaster help page on sitemaps:
Sitemaps are particularly helpful if:
- Your site has dynamic content.
- Your site has pages that aren't easily discovered by Googlebot during the crawl process - for example, pages featuring rich AJAX or Flash.
- Your site is new and has few links to it. (Googlebot crawls the web by following links from one page to another, so if your site isn't well linked, it may be hard for us to discover it.)
- Your site has a large archive of content pages that are not well linked to each other, or are not linked at all.
Enjoy,
Robert C. Cartaino
Robert Cartaino
2009-07-08 17:20:40
What about when a user deletes a question? Is there ever a time that they would regenerate the entire sitemap?
Andrew Garrison
2009-07-08 17:35:34
No need to update for a deletion, as long as your site returns a 404 on that question. Google'll ignore the 404 and remove it from index, so no harm done.
ceejayoz
2009-07-08 17:36:38
Does this mean SO's sitemap items won't ever get the lastmod field updated? How will the search engines know when to reindex a question page?
Lars Haugseth
2009-07-08 17:37:30
why can't I find stackoverflow's sitemap? It is clearly stated as "sitemap.xml" in the robots.txt, but that file does not appear to exist:http://stackoverflow.com/sitemap.xml
Andrew Garrison
2009-07-08 17:39:58
lastmod is a stickier issue, which is why I'd use the technique posted in my answer - generate it out of the database on-demand. StackOverflow may do this - they block access to sitemap.xml for non-Googlebot user agents, so presumably there's a load on the server from accessing it.
ceejayoz
2009-07-08 17:40:56
@average - If you spoof a Googlebot user agent, it shows up. They block it for normal browsers.
ceejayoz
2009-07-08 17:41:30
It's funny, now that I think about it, I actually read that article at codinghorror a while back, but I completely forgot about it. I must've read it before my morning coffee....
Andrew Garrison
2009-07-08 17:50:20