Challenges when Designing Large Websites?

Question

Possible Duplicate:
What should a developer know before building a public web site?

What are the challenges that are commonly encountered when designing and implementing a large site like Stack Overflow?

Answer 1

The largest challenge is probably maintaining it. As long as you didn't release anything you can debug at will. As soon as something goes life, you'll have to make sure that users don't see any debug data. So you'll have to setup a shadow debug instance of your website. The fun starts when a bug only occurs in the live version, not in your shadow version ;)

Answer 2

To build this site, you need:

-database expertise - how to design the tables, create the right indexes

-some advanced knowledge of Javascript to do the front end (fuzzing out the uninteresting questions, the tabs on the profile pages, the push alerts to let you know when a page you are looking at has changed, etc.)

It's actually not all that challenging to create this site, even with a lot of users. All the open source databases scale these days when set up properly. Heck, even CraigsList runs MySQL.

I'm pretty sure StackOverflow runs on a single server to date. At the point at which you get a bit larger, you also need to know how to shard your database in some reasonable way.

Answer 3

concurrency - you will get thousands of accesses in one same second - so the solution is enabling proper caching (os or server level - memory/files)

Answer 4

In my experience - and I think Jeff has said as much - the problems with writing a large-scale, public facing, high-traffic site is not a technical one. Solid scaling architecture, maintainability, language issues - they're all fixed by getting good programmers, that's not the problem.

What's really hard is planning for and solving the user experience issues, especially the social ones. Developer experience helps, but new products create new problems and what the developer feels is the right solution may not be the same as what the audience wants.

Answer 5

One of the things experience has taught me (and continues to teach me, as I am slow to learn this lesson apparently), is that more features is not always better. I find it a challenge not to add every single idea that pops into my head. If you head over to meta.stackoverflow.com, you'll notice many threads about feature requests or tweaks, and also how many of them have been turned down.

It takes a bit of foresight and a focus on where you want to take the site to turn features down. Improperly adopted "features" can muddle the user experience and dilute the intent of the site. Not to mention bogging it down with buttons, form fields, animations, etc.

Answer 6

In addition to the application's structure, a solid source control environment with branching, build environment, testing environment, and a stable deployment and rollback process. Probably often overlooked is a disaster recovery process.

More importantly, a development process that communicates the intentions of the design effectively to the developers and the testers. With that, a honest and open feedback system to make sure people are communicating. Human communication is one of the toughest issues.

Answer 7

Pure infrastructure requirements. Take a read sometime on Google about someone like Facebooks IT backend. Its really incredible.

Answer 8

Not overanalyzing and producing answers to things that don't have questions. When you have a real problem fix it.

Answer 9

Good developers can give you one answer. Good designers can give you another.

Marketing, and actually getting people to use your shiny site? That's the hardest part for a great many groups.

Answer 10

stackoverflow is not a particularly "big" site from a feature perspective compared to ebay. It's an extremely elegant site that hosts a tremendous amount of useful content via a well considered and limited feature set. My advice is to draw lessons from the craftsmanship of sites like this rather than their "size".

Answer 11

I was going to say that attracting an audience is the first challenge to building a large site, but Dean beat me to it and it sounds like you're more interested in the technical aspects anyway.

If you have the time to browse and dig and reflect, the stackoverflow blog has lots of good stuff about the building of stackoverflow itself; I don't think SO is quite in the same weight class as Amazon and eBay, but I'd definitely consider it to be a "large" site/app by almost any measure. So this isn't a bulleted list of things to study, but the blog does have some interesting examples of how those sorts of issues can play out int the real world. For example, Podcast 65 talks about "lessons learned" during the first year of development.

Answer 12

In addition to the technological hurdles, there is always the challenge of getting A LOT of strangers to visit your site and participate. That, in my opinion is the hardest part. How do you create something in such a way that causes people to want to use it?