How is AWS for Data mining for school project?

Question

I have to do a class project for data mining subject. My topic will be mining stackoverflow's data for trending topics.

So, I have downloaded the data from here but the data set is so huge (posts.xml is 3gb in size), that I cannot process it on my machine.

So, what do you suggest, is going for AWS for data processing a good option or not worth it?

I have no prior experience on AWS, so how can AWS help me with my school project? How would you have gone about it?

UPDATE 1

So, my data processing will be in 3 stages:
1. Convert XML (from so.com dump) to .ARFF (for weka jar),
2. Mine the data using algos in weka,
3. Convert the output to GraphML format which will be read by prefuse library for visualization.

So, where does AWS fit in here? I support there are two features in AWS which can help me:
1. EC2 and
2. Elastic MapReduce,
but I am not sure how mapreduce works and how can I use it in my project. Can I?

Answer 1

You can consider EC2 (the part of AWS you would be using for doing the actual computations) as nothing more than a way to rent computers programmatically or through a simple web interface. If you need a lot of machines and you intend to use them for a short period of time, then AWS is probably good for you. However, there's no magic bullet. You will still have to pick the right software to install on them, load the data either in EBS volumes or S3 and all the other boring details.

Also be advised that EC2 instances and storage are relatively expensive. Be prepared to pay 5-10x more than you would pay if you actually owned the machine/disks and used it for say 3 years.

Regarding your problem, I sincerely doubt that a modern computer is not able to process a 3 gigabyte xml file. In fact, I just indexed all of stack overflow's posts.xml in SOLR on my workstation and it all went swimmingly. Are you using a SAX-like parser? If not, that will help you more than all the cloud services combined.

Answer 2

Sounds like an interesting project or at least a great excuse to get in touch with new technology -- I wish there would have been stuff like that when I went to school.

In most cases AWS offers you a barebone server, so the obvious question is, have you decided how you want to process your data? E.g. -- do you just want to run a shell script on the .xml's or do you want to use hadoop, etc.?

The beauty of AWS is that you can get all the capacity you need -- on demand. E.g., in your case you probably don't need multiple instances just one beefy instance. And you don't have to pay for a root server for an entire month or even a week if you need the server only for a few hours.

If you let us know a little bit more on how you want to process the data, maybe we can help further.