Do you know any large datasets to experiment with Hadoop which is free/low cost? Any pointers/links related is appreciated.
Prefernce:
Atleast one GB of data.
Production log data of webserver.
Few of them which I found so far:
Also can we run our own crawler to gather data from sites e.g. Wikipedia? Any pointers on how to do this is appreciated as well.
One obvious source: the Stack Overflow trilogy data dumps. These are freely available under the Creative Commons license.
It's no log file but maybe you could use the planet file from OpenStreetMap: http://wiki.openstreetmap.org/wiki/Planet.osm
CC licence, about 160 GB (unpacked)
There are also smaller files for each continent: http://wiki.openstreetmap.org/wiki/World
Few points about your question regarding crawling and wikipedia.
You have linked to the wikipedia data dumps and you can use the Cloud9 project from UMD to work with this data in Hadoop.
They have a page on this: Working with Wikipedia
Another datasource to add to the list is:
Using a crawler to generate data should be posted in a separate question to one about Hadoop/MapReduce I would say.
This is a collection of 189 datasets for machine learning (which is one of the nicest applications for hadoop g): http://archive.ics.uci.edu/ml/datasets.html