Hadoop: Disadvantages of using just 2 machines?

Question

I want to do log parsing of huge amounts of data and gather analytic information. However all the data comes from external sources and I have only 2 machines to store - one as backup/replication.

I'm trying to using Hadoop, Lucene... to accomplish that. But, all the training docs mention that Hadoop is useful for distributed processing, multi-node. My setup does not fit into that architecture.

Are they any overheads with using Hadoop with just 2 machines? If Hadoop is not a good choice are there alternatives? We looked at Splunk, we like it, but that is expensive for us to buy. We just want to build our own.

Answer 1

Hadoop should be used for distributed batch processing problems.

5-common-questions-about-hadoop

Analysis of log files is one of the more common uses of Hadoop, its one of the tasks Facebook use it for.

If you have two machines, you by definition have a multi-node cluster. You can use Hadoop on a single machine if you want, but as you add more nodes the time it takes to process the same amount of data is reduced.

You say you have huge amounts of data? These are important numbers to understand. Personally when I think huge in terms of data, i think in the 100s terabytes+ range. If this is the case, you'll probably need more than two machines, especially if you want to use replication over the HDFS.

The analytic information you want to gather? Have you determined that these questions can be answered using the MapReduce approach?

Something you could consider would be to use Hadoop on Amazons EC2 if you have a limited amount of hardware resources. Here are some links to get you started:

• hadoop-world-building-data-intensive-apps-with-hadoop-and-ec2
• Hadoop Wiki - AmazonEC2