Legacy Data Feed, trade off between Standards, Xml and Performance.

Question

We're working with legacy data feeds and apps that consume them. We want to introduce Xml but the additional performance overhead is hard to justify. How have you addressed this issue?

We're working with a number of pre-existing data feeds, often files in a well known directory which are updated every few minutes. One approach to making this legacy data standards compliant is to convert it to Xml and publish the XSD - making it available to all. However this means we're going to serialise/deserialise everything before we can use it, whereas currently apps just read the data.

We going from

File -> App

To

File -> Serialize to XML -> ESB/Network -> Deserialize -> App

The latter is clearly more structured and re-usable, a 'better' architecture. But the performance hit we're going to take is high.

Answer 1

Convert to XML only where you must; don't convert for the sake of "standards compliance" or "better architecture". Surely you'd rather be adding more features or otherwise improving your product, right? This is a good moment to reflect on the YAGNI principle.

Answer 2

You might consider a more "lightweight" data format like JSON. It requires far less effort to parse, and is convertible to XML by simple XSLT if the need ever arises to have XML somewhere.

We're currently using it in an embedded device project where WWW is only a tiny portion but we are very satisfied. Yes, JSON format config files, JSON over sockets bidirectionally, and quite importantly, JSON to static Javascript making a dynamic web app to control it.

Answer 3

Why have you not created a library for such clients? Maybe XML is okay in the "cloud" but on your own intranet with a legacy format I would just make sure a library was provided - My first intuition would be just wrap the format layer and the provider layer. It seems that most of the format layer is done (after all you have applications that use the data) and if (I admit a big if) the recovery/resiliency constraints are not too stringent, wrapping a file stream / socket stream is easy