Data Loading Software

Question

We deal with scientific research data and we have volumes and volumes of data put together in different template file formats (excel, csv, txt, xml etc). We were using old legacy C programs (developed inhouse) to load these data into our databases. (We use ingres as our DBMS). Are there any opensource software that is available for ETL (extraction, transformation , loading) process?. What have been your experiences, if you have used any?

Answer 1

Here is an open source solution for importing multiple file formats into a database system or other system type.

http://talend.com/index.php

At the company I work at we use SQL Server Integration Services which does similar things but it should come with SQL Server if you're using that.

Answer 2

There is an opensource set of bi and etl tools - have a look at Pentaho - I believe it's etl tool is called "kettle" - pretty rich set of functionality, gui tools for the etl process.

Answer 3

We use DBMS/COPY but it looks like it is no longer in production. It has a GUI interface for setting up scripts or you can hand-write them.

Answer 4

Based on what other Ingress users are saying, the 2 that are fairly well spoken of are Talend and Pentaho.

Pentaho site: http://www.pentaho.com/
Talend site - as already mentioned by Paul: http://talend.com/index.php