views:

1260

answers:

8

In my PHP application ( constructed using symfony framework and Propel ORM), when I add a record to the MYSQL database, I need to update an external MYSQL database using web service API provided by the external vendor.

The question is what is the best practice to maintain the database integrity. Take for example, if the first update is successful, and the second update is not, due to the fact that web service isn't available, I must be able to either

  1. Rollback the transaction for the first update, or
  2. cache the call to the web services and keep calling to the web services until the services become available
  3. Some other techniques that can maintain the multiple database integrity.

Specifically, I am looking for syntax like

void RootMethod()
{
     using(TransactionScope scope = new TransactionScope())
     {
        try
         { 
          SomeMethod();
          scope.Complete();
          CallWebService();
         }
         catch
         {
             scope.abort();
          }
     }
}

But not sure whether

  1. This is a good technique
  2. Or is this feasible in symfony as in C#

What do you think?

Edit: Some asked me why there is a need for two part update. It's because I am creating a front end application connecting to an existing back end application. And I don't want to change the back end application. So inevitably there will be some overlap. Therefore there is a need to synch the data

Another Edit: The two parts transaction must be done together, doing a cron job to synchronize the table is not desirable

+3  A: 

This is going to be tricky. You need 2 phase commit for a reliable solution, but that would be a ton of work to implement for your specific needs.

Maybe a great solution is not actually called for. Are you under difficult performance constraints? Generally transactions should be short in time... but maybe you should keep the transaction open around the webservice call? This would reduce the overall throughput of the database (at the least)... but that might be perfectly acceptable.

The approach you showed will have problems handling hard system failures (power failures, hardware faults, etc). To work around that you'll need to add tracking to your main database and a background process/startup process to handle failures. Pretty fiddly to do, but certainly possible.

Some failures might end up not being fixable (first part succeeded, second part failed, first part can't be undone because another transaction has updated the same data). It all depends on your precise business rules. An accounting system would be the easiest because undo'ng a transaction is actually done as offseting records and not updates.

Good luck.

Keeping the transaction open while you do the web service call still leaves windows where the state is inconsistent. There is a window where the local transaction rolls back, but the web service transaction completes.
janm
+1  A: 

Keeping databases in sync is a hard task, depending on what data you have, can you add another table that contains what has changed and then run a separate script by cron or separate code that attempts to update the web service and bring it in sync with the changes that are stored in the database. If the changes are successful it removes the flag specifying that the change has not been sent on to the remote server.

On the local database after the data has been inserted you could use a flag to specify that it should not go live yet, and any and all data that is then synced causes that flag to be changed to fully committed.

What is the specific reason for doing this? Why do you need to keep two databases in sync in the application itself, are you able to sync it every hour instead?

This will require state tracking of the data and whether or not it has been committed successfully to both ends.

My personal choice would be number 1. Roll back the local transaction, unless that is absolutely not possible use number 2.

X-Istence
*Why do you need to keep two databases in sync in the application itself, are you able to sync it every hour instead?*It's because I am creating a front end application to an existing back end application. So inevitably there will be some overlap. Therefore there is a need to synch the data.
Ngu Soon Hui
+1  A: 

I didn't understand, is your app in PHP or in C#. If it is in C# (WCF) and the web service is a WCF (or supports WS-AtomicTransaction) then this is possible.link text

link text

kanad
It's PhP, not C#
Ngu Soon Hui
+4  A: 

The big question is whether duplicate updates to the web service matter, and if they do whether they can be detected. If you can detect duplicates (usually with a unique transaction number) or if duplicates don't matter, then you can build a two phase commit style approach which is reliable.

If duplicate transactions to the web service cannot be detected and the updates are not idempotent then you are out of luck.

This is the basic algorithm:

begin transaction;
do local work;
save information for external call;
set an appropriate time for next attempt;
mark external call as not performed;
commit work;

begin transaction;
make external call;
if successful
   mark external call as performed (or delete the record)
else
   set the time for the next attempt
commit;

You then need a regular task, thread, or whatever that does something like this:

for each record where the time for the next attempt <= now
    begin work;
    if the remote service has not performed this transaction
        make the remote call;
        if successful
            mark as done;
        else if too many attempts
            mark the transaction as permanently failed,
                   alert operator;
        else
            set the time for the next attempt;
        endif;
    else
        mark as done;
    endif

    commit;
 endfor

This approach handles all the failure conditions reliably, and ensures that both pieces of work are eventually done.

The basic failures:

A failure before the first commit completes: Everything rolls back.

A failure after the first commit but before the web service completes (this includes transient failures in the web service itself): The remote webservice transaction is replayed by the recovery task.

A failure after the web service completes but before the second commit completes: The duplicate web service transaction is detected by the recovery task and the local record is dequeued.

Failures in the recovery task: Essentially the same as failures in the second transaction.

Other notes:

  • A gradual back-off approach is useful for failures. If there is a transient failure on a service you want to slow down your retries.

  • If you have an ordering requirement on the external service you might need some additional structure.

  • Depending on how you implement the recovery task, you could just leave the web service calls to that task and not have the second transaction in the main application flow.

Response to the additional requirement: "The two parts transaction must be done together, doing a cron job to synchronize the table is not desirable"

My reading of this requirement is: "the two systems should never fail."

When one of the systems (or both) of the systems you need something to pick up the pieces and reconcile things. You can use a fully fledged TP monitor to do transaction co-ordination, or you can build a simple monitor like the one in my example that handles your specific case. Either way, there is something that keeps track of what was happening so that things can be resolved correctly after a failure condition.

If your requirement really is that things always happen together (and a transactional message queue or two phase commit approach does not work for you), you would be better off storing the data for both systems in the same database (aka "resource manager") and having a single resource manager transaction.

If you do get a solution to this problem that meets the requirements of having two separate systems consistent across multiple transactions and never requires subsequent reconciliation after a failure, you should write it up and get it published in The VLDB Journal , ACM TODS or IEEE TKDE.

janm
+2  A: 

I don't think rolling back is really going to help the situation. If your web service is down, making more calls is only going to compound the matter, and then you have to worry about whether or not your rollback went through etc.

I would do this with scheduled full syncs. What is your margin for error? Are you willing for the database to be slightly out of sync? By how much? Would it be a big deal to have a synchronizer run every night or so to fix any issues that cropped up? How often is the web service down that you have to worry about this?

The update queue for failed web service calls is a decent idea, but if your web service is down, there's probably going to be a ton of those at once, not just one or two, so you might as well do a full sync after an outage anyway.

Really your answer depends on those questions. Don't make the mistake of assuming your entire program will crash if things are 0.01% out of sync for 10 minutes. Figure out what acceptable margins for errors are.

UltimateBrent
That's a good answer.. but supposed that I am only willing to tolerate for out of sync situation for 10 minutes?
Ngu Soon Hui
I assume the "update queue" is my answer. "You might as well do a full sync" really depends on how much you're dealing with; that doesn't scale.
janm
"only tolerate an out of sync situation for 10 minutes": Take the universe back to your place of purchase for a full refund. What happens if the web service dies before is acknowledges completion and stays down for an hour? You don't know if it committed or rolled back until it comes back up.
janm
If you're only willing to tolerate an out of sync situation for 10 minutes, then you could still do "full syncs" but it really depends on your data. Like janm said, it's not scalable. If you've got a lot of data, and need it in sync all the time, you'll have to do the update queue.
UltimateBrent
+2  A: 

Do not try to build 2-phased transaction management logic yourself. You will get it wrong, trust me. If it is available in your proggy environment as it is in C#, then use it. If not, then don't go building it yourself.

In most cases, it is easier to design a system with distributed transactions across multiple online databases, but it is easier to live with the operational aspects of a system if it includes a queue for (1) resilience in the face of network unavailability, (2) consistent latency behavior in the face of high loads.

so, confine your transaction to local resources:

make a change to reliable store (I believe this would be called a "resource" in X/Open transaction parlance)
enqueue a record of that change in a disk-backed log

then, on a schedule (every hour, every day, whatever)

while not done
    pop item from queue or log
    sync that change with the external, remote resource

At times of high load, your queue will fill up, but the network load and transaction latency will stay relatively constant. It's a little like the monthly-budget plan for your home heating bill. At times of relatively low load, the queue will drain.

Cheeso
+1  A: 

Maybe you could try not to put this logic of multiple updates in your application, but use an external process that knows what to update when and what to do if an update fails. Oracle BPEL for instance is such a process. You can configure it to orchestrate different services, for an example see http://alisonatoracle.blogspot.com/2006_01_01_archive.html

Might be overkill for your app though, depending on it's size...

Jack
+1  A: 

Is there a specific reason the back end database must be updated at the same time as the front end?

If not, one approach would be to update the front end as your database-of-record and mark the updated record as requiring synchronisation. A housekeeping task can periodically pick up all records from the front end that are marked as requiring synchronisation and then update the back end. When the back end has been updated, clear the synchronisation flag on the front end.

Andrew