How do you do system integration?

Question

I curious to how different people solve integration of systems. I have a feeling that the last years more and more work has gone into integrating systems and that this kind of work need will increase as well.

I wondering if you solve it developing your own small services that are then connected or if you use some sort of product (WebSphere, BizTalk, Mule etc). I'd also think it'd be interesting to know how these kind of solutions are managed and maintained (how do you solve security, instrumentation etc, etc), what kind of problems have you experienced with your solution and so on.

Answer 1

wow - Ok - will get a post on this but will be big.

Intergration needs to be backed up with a big understanding by the business on the benefits - Get an opertating model sorted out - as the business may acutally need to standardise instead of intergrate, as this can be costly - its why most SOA fail! Enterprise Architecture: Driving Business Benefits from IT Author(s): Jeanne W. Ross

If intergration is needed you then need to settle on type of integration.

What are the speed and performance metrics?

We have a .NET SOA with a Composite Application that uses BizTalk 2006 and webservices with Line of Business Applications. Performance of the application at the composite end (consuming) - is limited to the speed of the webservices (and their implementation) in the line of business application! We need sub <3 second return on results - list of cases. Could not be acheived in the webservices so we need to get go to the database directly for initial search. Then over the webservices for case creation. Cost implications and maintance becomes an issue here.

The point here is to look at the performance criteria in the specs and business requirements this will help in look at the type of integration that you need to do - WebServices (HTTP), File Drop/ EDI etc

Functionally for intergration you need to then look at the points of failure in the proposed architecture - as this will lead to a chain of responisblity in SLA/OLA. You may need to wrapper the intergration/faliure points into things that you control.

On similar point about integration with Line of Business is with how much do you need to know about the other product before you can integrate? Yeah Webservices are supposed to be design by contract but the implementation is often leaky and you need to understand alot about what is happening - and if this is a product that you dont control the abstraction even with webservices leaks into your intergation technology aka BizTalk.

Couple these two points together and you the best advise is to get a intergration hub type like BizTalk - wrapper the line of business applications in webservices you create - so the BizTalk side can be free from leaky abstractions then you also can reduce the points of failure as the you have decoupled the line of business application from the intergration hub and the point of failure to a single source rather than inside an orchestration.

Instrumentation and diagnosics in SOA and Intergation Porjects are hard to acheive! - Dont let any shiney sales person try and tell you differently! Yeah MOM with MOM Ent can do this UniCenter can do blah.

The main problem is understand what the error aka burps in the intergation mean and how to recover from them... You end up with messages stuck and you need to understand what that means to that busienss process. You can get an alert to say - processers are 100% Ram 100% orchestrations have failed - but no real meaning. You have to engineer this stuff in to the solution from the outset - and hopefully into you points of failure.

Types of intergration patterns and how to do them do need to be considered too.

The above is a real world view of a .NET SOA with BizTalk in a LIVE implementation. But it is also due to the architectural limitations of this - BizTalk mainly is a HUB and SPOKE pattern.

Check out Enterprise Application Patterns by Martin Fowler

There are many ways to skin the task!

Other considerations... Platform/Developer Languages etc.

One of the big factors for us was the skills needed to start this stuff. We had OO devs with Java and C# understanding, but mainly C#. So we went for the MS stack. But when you choose the intergration type and the product to manage this they will need more skills in understanding that technology. But hey this is normall for us Devs right? Wrong many developers regardless of there expereince can come unstuck with the likes of BizTalk! Big shift in paradigm - which in part is due to messaging shift rather than code.

Best bit for last!

Numbers of transactions that are likely to be faced in the integration is probable the single biggest factor in all of this. As this will guide what pattern, points of failure and tolarance for such things.

You need to select best on anticpated volumes the right one. Something that can scale up and scale out! We selected BizTalk since it can scale up and scale out correctly and with better understanding than some others.

If you dont have volumes then look at not getting something to manage them and go for a webservice to webservice type style with no management - performance and failure understanding will need to be coded into them.

If your on windows platform with .net 3 take look at WWF/WCF as this can help in webservice to webservice - lots more in the acutal platform now for all these concerns without the overhead of BizTalk and others.

Hope this helps.

Answer 2

In my experience it depends on what kind of problem you are tacking.

In my experience it's difficult to beat BizTalk 2006 R2 for bang for the buck but it does imply the use of a Microsoft technology stack.

Websphere MQ seems to be an easier sell to larger corporates and it probably seen greater use at the enterprise level.

Both provide good instrumentation but it's really up to you as a developer to customize this to suit your customer's requirements.

In some cases I've found that a bespoke solution is most appropriate or leveraged technologies such MSMQ to keep costs down.

Answer 3

You mentioned WebSphere, BizTalk, Mule. Each of which has very different characteristics with its good and bad points. If just integration you are after, I will recommend Mule. I had very good experience with it and more important the architect is non invasive, so you could always migrate to a different ESB or other Buzz word complaint solution. One the the sweet spots of Mule is that it can be embedded within your application and your final artifact can be deployed on Webshpere, WLS, Glassfish etc... without even showing you embeded an ESB. Then this ESB can perform all the integration plumbing (managing connection types and protocols). Whereas some of the end points could be other integration solution you mentioned.

Answer 4

We are using Mule for a while (now investigate migration from 1.4 to 2.1.x version).

Well It's one of the best ESB with live community and quick reaction from vendor side, but IMO version 2.1.x is still a bit raw (or we are only the company that use it for calling CXF web :) see also my post for details http://www.nabble.com/Migration-from-XFire-to-CXF:-Is-Web-Service-Client-in-Mule-2.x-broken--to19969320.html#a19969320)

Answer 5

Hello,

we have an Oracle contract. So we are using the Oracle Stack. SOA Suite 10.1.3.4. Mostly BPEL solutions and for a simple transformations we try to use ESB.

The ESB has a bad Fault handling mechanism. For the BPEL there are many ways to handle errors. We try to develop java webservices to connect to the SOA Suite and our main systems are Oracle EBS systems. They communicate to legacy systems or other EBS enviroments through the default EBS Adapters that are shipped with the SOA Suite.

Problems we encountered is the lack off knowledge about the EBS adapters. We encoutered some problems with an BPEL solution that received information from the EBS systems. It was a hell of a job to get the solution production ready.

Securing our webservices isnt't a big issue. With the Oracle stack comes the Oracle Web Service Manager. With that we can secure, log etc. all the webservices.

The biggest problems we encounter is the lack of our own standards. Getting the business to feel that they can also build SOA solutions. We can't explain the benefits they get with a SOA solution. Faster? no ! Cheaper? Hell no! Easier solutions? Well, maybe when we get good reusable services ... well, that easier part has a problem within it: how do we know which applications use the reusable webservices?

We need an register, that can display this kind of information. Because we can't find a good opensource solution, we are trying to build our own register. Simple APEX solution, again from the Oracle stack. ;)

So someone knows a good product to register this kind of information?