Marking Changes to database...

Question

Hi guys...

I am developing application to be run in central server and distributed computers.

I am supposed to write application to backup the data from distributed machines and merge it in central server. I thought of compressing whole local database and sending it to server for merging. But as the database size grows the size of compress file also began to grow. So is there any way to merge data in central server without sending whole database. I need to do it on daily basis.

Daily take backup and send to server

Answer 1

Use an ETL tool, something like Pentaho Data Integration or Talend. You'll still have to think about how to identify the data you want to retrieve, but you get at least the following (from the PDI feature list):

• Rich transformation library with over 100 out-of-the-box mapping objects
• Broad data source support including packaged applications, over 30 open source and proprietary database platforms, flat files, Excel documents, and more
• Advanced data warehousing support for Slowly Changing and Junk Dimensions
• Proven enterprise-class performance and scalability
• Integration (EII), advanced scheduling, and process integration
• Unified ETL, modeling and visualization development environment for design of BI applications

It basically comes down to "it will work with whatever your data sources are, it will be robust and you'll be able to apply the expertise gained elsewhere".