What is a good hardware setup for programming concurrent and distributed applications?

Question

I don't have the money to build my own uber Blade system but I would like to get into concurrent and distributed programming (think CCR/DSS, Hadoop, Project Voldemort etc.).

I currently have a Q6600 with 4GB with some separate hdds but that's about it. While I can write multi-threaded programs I can not properly test distributed filesystems / key-value stores and look for associated bottlenecks (disk access, network, etc.).

Does anyone have some recommendations? Buying some small cheap boxes and setting up a mini network? Or maybe a single box with two i7's and ESX and a simulated network?

edit:

I'm currently using VirtualBox and VmWare and this does not look good enough for me, correct me if I'm wrong: The hard drives could lock for instance, either because two virtualized machines run on them, or because all hard drive access is channeled through the same hdd controller. The network is entirely virtual, so no real case test here either.

If I go the virtualization route, what would you recommend so I can get as near to 'real-life' as possible?

Answer 1

Virtualise for your distributed system tests. It's much easier to 'pull the plug' on a machine, disconnect the network cable etc.

Sun VirtualBox is an excellent free virtual machine which I've found extremely convenient for development purposes. Most of it is also Open Source, if you're into that.

---

As for the multi-threaded part, it's actually easier - always test with more software threads than your number of hardware threads. And then, just for fun, do something like writing a 10 GB file to your hard disk, plug/unplug hardware to throw the scheduler off. You'll get surprising results.