how does StackOverflow optimise the performance for the display of the questions?

Question

Hi there,

i am trying to learn c#.net to program a web app.

And having learned that stackoverflow uses C#.net I am happy to discover it.

I noticed that at the home page or at the questions section, whenever i refresh the page. The page always returns me the latest information without fail and at acceptable speeds.

I am not sure how do you do it. Sorry for the long series of questions. I am trying to learn what is the best practices for data retrieval, paging, performance , etc

I know that the homepage only returns a limited number of questions and their stats but the questions section actually returns everything.

How do you optimise it?

1) For the homepage, do you always grab ALL the stats of the recent questions? so your query is something like "select * from questions order by datetime_created limit 20" ?

so the * contains ALL the info including question title, id, views, etc?

Do you use HttpContext.Current.Server.cache to help with that?

2) For the questions, this is even more intriguing.

how do you do the paging?

Do you always grab from the database only the results for the specific page?

or do you grab all the results and store it into a dataset? Then you use some kind of datagrid control to help with the paging?

If it is the latter, how do you maintain the data to be updated?

Thank you.

Answer 1

SO uses MVC and LINQ2SQL. I'd listen to some of the podcasts to get more of an idea of the specifics. I do know they use a lot of caching but not sure if that includes the question list on the home page.

Answer 2

_{I have no idea how they did it - I didn't write SO.}

For something like this, I'd use some sort of caching mechanism for the entire Question class with all of its Answers. The cache would be short-lived, but since new/hot questions are viewed very often they would stay alive. Older questions would have to be requested from the DB. This would also prevent threading problem when one person answers the question and while another thread views the question.

Another thing you can notice here, is they make heavy use of AJAX. But since AJAX.Net is extremely bandwidth-wasteful, they implemented the AJAX calls so they would return simple JSON objects e.g. when upvoting only a success object with a new number of votes or an error message is return, for example: _{(this is a made-up example and is not representative of what happens because I can't be bothered to check right now)}

{"status": "ok", "votes": 3}

AJAX.Net would return the entire UpdatePanel's contents, whatever its size, which no matter how small, would still be quite large.

Answer 3

Here on Stack Overflow, we try to use aggressive caching on many levels:

• pages entirely cached by IIS' output cache, regardless of user authentication
• pages cached only for anonymous users; registered users see the most recent content
• portions of pages' html cached for everyone; HttpRuntime.Cache is used for this

The home page is made up of three cached html pieces - recent questions, recent tags, recent badges - each with a different duration.

A questions list page will cache the ids (Int32[]) of all questions for a particular sort/tag filter, making paging trivial. Further caching on the stats (e.g. question count, related tag counts) is done, as well.

A question detail page will be entirely cached for anonymous users, while registered users see the latest goods. Also, the related questions on the side are cached to disk for a longer duration.

While we try to cache entire pages wherever possible, we do show user information at the page top - some parts just cannot be cached.

So look at caching like a puzzle - what parts can be safely shared between all my requests? Based on expense, what parts MUST be shared across all my requests?

Answer 4

You may also want to take a look at this Stack Overflow Inspired Knowledge Exchange.

It's a group of articles trying to emulate Stack Overflow. You should be able to find the codebase here at CodePlex.