Integration transport choice (Oracle + SQL Server)

Question

We have several systems with Oracle (A) and SQL Server (B) databases on backend. I have to consolidate data from those systems into the new SQL Server database.

Something like that:

(A) =>|---------------|
      | some software | => SQL Server
(B) =>|---------------|

where some software is:

• transport (A and B systems located in the network)

• processing business logic (custom .NET code)

Due to first point, I need some queue software or something similar (like MSMQ, Service Broker or something). In another hand, I can implement a web-service instead of queue.

(A) =>|---------------|-------------|
      | queue/service | custom code | => SQL Server
(B) =>|---------------|-------------|

The question is: which queue/transport framework should I use with Oracle and SQL Server databases?

It would be nice, if I can post messages to MSMQ in both Oracle and SQL Server stored procedures (can I?)

It would be nice, if I can call a web-service in both Oracle and SQL Server stored procedures (can I?)

It would be nice, if I can use something similar in both Oracle and SQL Server stored procedures (what exactly?)

What software should I prefer to my requirements?

UPD: some techspec

This would be a regular sync process. Once a day I think.

Latency is not critical (>0.5-1 hour is ok).

Amount of data: 1-50 MB per sync from each system.

Encryption is required while transfer.

Answer 1

I would suggest creating an SSIS package that transfer the new data from the server A,B to the new server when invoked. You would launch the SSIS package on a schedule, say every 30 min, from the new server.

If both A and B would be SQL Server then Service Broker would make sense in order to provide a very low latency. But with one of them being Oracle, and with no real-time requirements, it looses its appeal. As a side note, you can see here an example of using Service Broker for High volume real time contiguous ETL.

Doing the transfer as an SSIS package makes for ease of maintenance (you can modify the package with relative ease), it does not require invasive changes in the existing system, is quite performant and there is a large tonne of SSIS know-how available online.

I would advice against using MSMQ for several reasons:

• when transactional reliability is needed you'll have to involve all MSMQ related operations into distributed transaction (DTC between the MSSMQ dequeue and the SQL server insert/update on the new server) which will slow doen the processing throughput significantly
• You'll need to come up with quite a few lines of code for the marshaling/unmarshaling and shredding the deltagram messages into the target system (I know codding is fun, but SSIS is simply better at this kind of jobs, and easier to maintain)
• MSMQ limitation of 2GB per queue is quite small in real world (fills up quickly if your traffic increases and you have a maintenance downtime)

The real problem I'd be worried about is how to detect changes on A and B: when the SSIS job comes every 30 minutes, how does it know what data is new? Specially, how it detects deletes...

Answer 2

My company, Temboo, is a SaaS provider that specializes in automated and on demand data integration and synchronization via our visual programming language. We can help you create the synchronizations or allow you to create them fully yourself. We empower our customers to build their own solutions and edit any data flows that have been created for immediate flexibility. Let us know if we can help.

Luke