How would you go about building distributed internet-scale systems ?

Question

Hi,
There are many job descriptions having a phrase similar to "building distributed internet-scale systems" in the list of required skills. Although I do have a basic sense of what it probably involves (using technologies like Hadoop, key-value stores, message queues, focusing on scalability, availability etc), could you describe how would one go about doing it ? Any resources to read upon this would be great.

Thanks,
Abhinav

Answer 1

You really answered your own question. Choose a distributed technology like Hadoop and build software with it, blog about it, do whatever you have to do to become an expert in the area. If you don't like Hadoop, there are many other tools that you can choose from.

If you're looking for actual software needs, there are plenty of opportunities around.

1. My friends used to work on the Netflix challenge using Hadoop (dealing with an extremely large database) that is still fun to work on even though the challenge is over.
2. Nagios is in need of some fresh new ideas, why don't you apply some of the distributed concepts that Hadoop offers to the Nagios system? I actually have a post on SO that I'm trying to get people involved on here: http://stackoverflow.com/questions/3251856/needed-software-that-uses-hadoop-nagios

If the job postings requirements are throwing you off, maybe you should talk to the recruiters more and get an idea of exactly what their requirements mean to themselves. It is never going to hurt you to apply and have the opprotunity to talk with their developers to see what they really need. Heck, everyone has got to start somewhere.