persisting persistence; or moving legacy webapp to non-relational store

Question

Suppose you have a huge app with a data access layer bound to SQL You want to provide other non-sql DAL, for instance a GoogleAppEngine instance or XML-based backup. How would you approach this migration to a different storage paradigm but with limited flexibility, since it is legacy.

I might be missing info here, but One solution I have in mind is to first rewrite the current legacy DAL into a standard API base, for instance JDO; from there, interfacing with any storage would be a matter of time based on the technology-specific learning curve.

AND OR NAND?

Answer 1

A lot depends on the specific details, but from what you describe I would probably start by building a Data Access Layer with DAO's that explicitly define an interface and an implementation to the current store.

Once that is up and running you can start creating an implementation of the interfaces to your new storage and migrate step by step.

Answer 2

In theory, as you also mentioned, you just need to use some standard interface like JPA, and change the implementation or the storage behind it. Unfortunately, in practice this doesn't work so smoothly. It can be done, but you're surely gonna spend some time on issues you just didn't expect.

I would start by stabilizing the service interfaces, and migrate them feature by feature. Start with the easiest one, something that is small enough and separated from the other parts of the system.

However, as a first step, it's not a bad idea to rewrite your code to use some standard interface. JDO is known to handle non-relational databases somewhat better, so you might benefit from choosing it.

As for Google App Engine, most probably you will need to change your data model entirely, if you have already designed it with an RDBMS in mind. Just to mention a few constraints, there is no JOIN, and you also cannot use OR in your queries. So I suggest you to experiment with it before you jump into some huge migration project.

Answer 3

DataNucleus, as used by GAE/J for BigTable, also supports persistence to XML ... using JDO. Consequently it would be trivial to support persistence to that too. You have limited joining capabilities in GAE/J ... not a limit of JDO, just of how they have implemented such a thing in the GAE/J plugin.

--Andy (DataNucleus)

Answer 4

Indeed, GAE/J does have a limited JDO api (0) ; their use of the JDO naming is misleading ... :(

On the other side, as you suggested, JDO does support Joins, as annotations (1) and/or as metadata(2); which can be retrieved afterwards using a JDOQL query.

Please correct me if I am wrong, but JDO\JDOQL feels less powerful than an SQL query. The power of SQL is the capability to join on the fly a multitude of related tables. I still cannot see how can I support such a query, like this simple example from my webapp

(human) schema: a product line has many catalogs, which have many products, any invoice has any of those products

SELECT a.id, a.name, b.id, b.name, c.id, c.name  from products a
left join catalog b on b.id = a.fk_catalog_id
left join productline c on c.id = b.fk_productline_id
where a.id in (select distinct product_id from invoice_product where invoice_id = <PARAM> )

links:

theweb@(0); code.google.com/intl/fr-FR/appengine/docs/java/datastore/usingjdo.html#Unsupported_Features_of_JDO

theweb@(1); www.datanucleus.org/products/accessplatform/jdo/annotations.html#Joins

theweb@(2); www.datanucleus.org/products/accessplatform/jdo/orm/one_to_many_list.html#join_uni