Agile Development and ESBs

Question

I'm working on shifting our corporate technological paradigm to Agile Development. It's been a hard process but we're almost there! :)

We have legacy systems for our database management (used to be Access, now ported to .NET and MS SQL) and we're developing a framework for our future vision. We want to migrate as much as possible to the web. But we want to integrate the current system with the "upcoming" one. We won't be overlapping tasks and functionalities.

My vision is to move all contact info on our users to a different database, linking those "profiles" back to the MS SQL for their history and accounting info. We'll keep all accounting systems on the desktop app, but there's a lot more functionalities that we're about to add that will rely heavily on the web, especially Ruby on Rails.

I guess the question is: why ESBs? Is there a way to create a SOA without going gung-ho with complex ESBs systems. The whole idea is to K.I.S.S. anyway. Can a SOA be created in a way that allows the desktop/web/mobile to be interfaces, keeping the functionalities on the business logic (of course some functionalities would have to be implemented on the interface, but keeping that to a bare minimum). And do ESBs even fit an Agile philosophy? The more I read and study them, the less I think so! :/

Thanks for your input folks! If you need me to clarify, just make a few questions and I'll do my best to do so! :)

Answer 1

ESBs fit well with agile once the framework/infrastructure is in place. You'll find that you can create a new system in pieces, run the new pieces in parallel with the old system for a while, and gradually turn off the old parts of the system until only the new system is left, and no one will ever know the difference

a basic SOA just defines services instead of applications; an ESB manages the services in channels to hide the endpoints, making upgrades and substitutions much more "agile"

Answer 2

The whole migration is what got me to ESBs... But the whole idea of an ESB seems way to complex to solve a problem that involves around 30,000 profiles! We are on the verge of some exponencial growth (to a few million profiles) and maybe starting on a new path would be best. How easy is to link an entry that sits on a MySQL DB to data stored on MS SQL DB? I don't want double-entry obviously, but there might be a more agile way than a "whole" ESB... I do understand that a SOA with an ESB can be pretty agile in terms of upgrades and substitutions but would it be an overkill? :)

Answer 3

I've learnt fairly quickly to shy away from the term "ESB" as itis very overloaded and means different things to different people (and at times different things to the same person :-))

The key thing, naturally, is to ask yourself what it is you actually require.

Wrapping your database(s) as a service is likely to be a wise choice, especially if you have multiple client for this data; you will have to spend a good amount of time thinking about your contracts and scoping, but agile can greatly help here.

The question is now how do these service get called, and I think you need to weigh the likelihood of the clients and services change and how your system is going to evolve.

A service bus helps masking the services from their client (which can be other services) and this "masking" can relayte to location, protocol, formats, codes, etc. some forms of a service bus also maintain the itinerary (what needs to be called, and when) but I generally dislike the idea.

so - the question you need to ask yourself first, I think, is what do you require to start with and how much up front investment you are wishing to make (and can justify)

For example, if initially you are happy with a more point-to-point approach, your clients can call the service directly; at a later stage, as the service evolves, you can introduce the "middle man" to broker the request and response (yes - you can call it ESB if you like).

Alternatively you can start with a basic "middle man", so that the clients never call the service directly, but have just the features you need to begin with and expand it's capabilities as requirements form; it may well start as being a simple forwarding machine.

Ideally you would build on top of a product that has many capabilities built in; BizTalk Server is a good matchif you're on MS stack (but has it's learning curve)

so - apologise if this is not a very concrete answer - but I guess my main point is that "ESB"does not have to be an overkill, it simply comes down to what you wish to have at day one, and agile (and SOA) definitely helps by allowing you to gradually evolve rather than anything big-bang like.

(if anything above is complete nonsense or just a bit unclear it's due to lack of sleep with a new born in the house! apologies :-) )