Should we introduce BizTalk/ESB?

Question

My company are about to implement a new architecture in which we have proposed BizTalk (we are a Microsoft shop) as the Enterprise Service Bus (ESB) in a SOA (please don't quote Service Oriented Ambiguity) environment.

Our business is to take Orders through our new Order Capture GUI which must connect to our Customers Database, Product Catalogue, Ordering System and some other ancillary systems each of which will be exposed as WCF services, orders are then passed to our Order Management and other downstream systems for fulfilment and finally to our Billing system for invoicing. Currently each system has its own GUI and uses manual process to pass information between them, in an effort to automate and integrate the natural thought was to introduce an ESB to connect them.

Some of my rationale for an ESB is, the bus will worry about how to connect the systems (each system is agnostic and knows nothing of any other system) and how to format/translate the information. It is highly likely that in the future some of the existing systems will be swapped out for new systems or systems within our family of companies.

This seems to make sense to me but I am now being met with some resistance as to why introduce it when a Point-to-Point solution could suffice.

Unfortunately in the company history (prior to my appointment) an initial attempt to introduce BizTalk failed, but I am confident that it has a place and I can deliver it.

My question is perhaps not so much about BizTalk but whether an ESB is a good idea in my scenario described, when does it make sense to introduce an ESB?

Answer 1

I think it makes sense to have a data broker based on the requirements you described, but I personally don't think that BizTalk would be the best choice in your situation. For the type of integration you've described, I would look at a hardware data broker appliance. Some that I've seen work well are IBM DataPower, Vordel's appliance, and Layer7's appliance. For the type of calls you'll be using this for, an appliance is ideal. They provide routing, transformation, and translation, plus they'll protect against things like schema poisoning. They'll also handle authorization, authentication, and auditing by linking it to your user store (I'm guessing you have an Active Directory user store based on the environment you described, but it will also work with LDAP)

An appliance will be BizTalk or any other software solution in terms of cost of ownership (no support costs), and the performance of any appliance will likely beat BizTalk by an order of magnitude.

Answer 2

Okay. ESB Guidance on Biztalk from the presrciptive architechture group - http://msdn.microsoft.com/en-us/library/cc487894.aspx

We use BizTalk where I work to do a lot of things. He have some simple point intgerations. We have some more complicated point integrations with hihgly customized adpaters and pipelines. We have divisional enterprise system integrations for customer master, product info and price and quote to order. These are all seperate BizTalk applications. Some quite small and some quite large. We mainly have used BizTalk to do point to point/many point slutions without using an ESB pattern. The implementation of an ESB implies a level of governance of the bus itself and the enterpise message standars that will be permitted on the bus. If you will be interfacing with a large number of systems with a large number of different formats - ESB makes a lot of sense. If the integrations you want to achieve are less ambitious, an ESB may be overkill. That being said, it's a clean and extensible architechture. You'll have to make the cost value decision.

BizTalk is also a complicated beast but with allof the moving parts comes a level of flexibility that is wonderful. But be prepared for a learning curve or some consultant costs.

Answer 3

I tend to avoid the ESB term as I believe it is grossly overloaded; at the end of the day, in all the various descriptions I've heared, it is just a pattern, once which BizTalk supports very well.

So - do I think BizTalk will fit what you wish to do? categorically yes. Do I think you're right to avoid point to point connections - also yes, but, like with any refactoring exercise for reuse, including any SOA initiative, you must consider how much change and now much re-use you expect to decide how far you're taking you're "decoupling" exercise.

Answer 4

It's a very distinct pattern. Typically when you are sending amessage from System A to System B, you do a direct conversion from the format of System A to the format System B wants. When you have an ESB, You convert System A's Message to the ESB Format (ie., generic PO, Order, etc.) and then into the Format required by System B. It's a two conversion versus 1 and also, the bus pattern requires every messag eto have a verb (ie. add, delete, update, etc.). This is a real important distinction and is what makes ESB very useful in integrations with lots of participating systems.

Answer 5

You need to talk about latency and throughput. Everything else is just bla-bla.

Answer 6

I just got asked this same question by a colleague and this is what I said to him:

In most integration scenarios you can go quite far before using something like BizTalk. I would make sure that I couldn’t do the integration more simply before going with BizTalk.

Only if it seems that the integration solution needs to scale to high volumes with low latency (it’s got a fantastic asynchronous publish-subscribe mechanism), and you need fault tolerance (it’s got redundancy, scaling and message retry features) and governance over the solution (it’s got Business Activity Monitoring) would you really have strong arguments to consider using BizTalk. And if you know that there are multiple future integrations that will be needed then it gets really compelling to use BizTalk.

On the other hand you need to make sure the skills are available to operate the thing. It takes a while to learn and a paradigm shift for the developers of the systems. However its built from the ground up in .NET and SQL Server so there is quite a lot of familiarity in the tooling and concepts.

I think the key thing is to get the conceptual architecture of a solution right taking into account the non-functional requirements like performance, availability, extensibility, resilience, robustness, and scalability and making sure they are properly addressed by the technical design. You may find its cheaper to pay the 35k$ per CPU license for what BizTalk gives you out the box than to develop for all these NFRs.

Also I've recently implemented the new ESB Toolkit 2.0 at a client and am very happy with it. The Itinerary (see the Routing Slip pattern http://www.enterpriseintegrationpatterns.com/RoutingTable.html) processing functionality really makes composing web services easy, flexibly and fast.