Locking problems with sqlite and SubSonic when using transactions on a single thread

Question

I'm getting locking exceptions when trying to use transactions with SubSonic and SQLite. I'm using this from a single thread and there are no other processes accessing my db, so I really didn't expect any such problems.

If I write code like this below, I get an exception on the second call to Save() within the loop - so the third call to Save() over all.

       using (TransactionScope ts = new TransactionScope())
       {
            using (SharedDbConnectionScope sharedConnectinScope = new SharedDbConnectionScope())
            { 
                SomeDALObject x = new SomeDALObject()
                x.Property1 = "blah";
                x.Property2 = "blah blah";
                x.Save();

                foreach (KeyValuePair<string, string> attribute in attributes)
                { 
                    AnotherDALObject y = new AnotherDALObject()
                    y.Property1 = attribute.Key
                    y.Property2 = attribute.Value

                    y.Save();  // this is where the exception is raised, on the 2nd time through this loop
                }
            }
       }

If I have the using() statements as above, or if I just have using (TransactionScope ts = new TransactionScope()) then I get a System.Data.SQLite.SQLiteException with message

The database file is locked

database is locked

The stack trace is:

   at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
   at System.Data.SQLite.SQLiteDataReader.NextResult()
   at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
   at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
   at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery()
   at System.Data.SQLite.SQLiteTransaction..ctor(SQLiteConnection connection, Boolean deferredLock)
   at System.Data.SQLite.SQLiteConnection.BeginDbTransaction(IsolationLevel isolationLevel)
   at System.Data.SQLite.SQLiteConnection.BeginTransaction()
   at System.Data.SQLite.SQLiteEnlistment..ctor(SQLiteConnection cnn, Transaction scope)
   at System.Data.SQLite.SQLiteConnection.EnlistTransaction(Transaction transaction)
   at System.Data.SQLite.SQLiteConnection.Open()
   at SubSonic.SQLiteDataProvider.CreateConnection(String newConnectionString)
   at SubSonic.SQLiteDataProvider.CreateConnection()
   at SubSonic.SQLiteDataProvider.ExecuteScalar(QueryCommand qry)
   at SubSonic.DataService.ExecuteScalar(QueryCommand cmd)
   at SubSonic.ActiveRecord`1.Save(String userName)
   at SubSonic.ActiveRecord`1.Save()
   at (my line of code above).

If I have the using statments nested the other way around, with SharedDbConnectionScope on the outside, then I get a TransactionException with message "The operation is not valid for the state of the transaction." Stack trace is:

at System.Transactions.TransactionState.EnlistVolatile(InternalTransaction tx, IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions, Transaction atomicTransaction)
   at System.Transactions.Transaction.EnlistVolatile(IEnlistmentNotification enlistmentNotification, EnlistmentOptions enlistmentOptions)
   at System.Data.SQLite.SQLiteEnlistment..ctor(SQLiteConnection cnn, Transaction scope)
   at System.Data.SQLite.SQLiteConnection.EnlistTransaction(Transaction transaction)
   at System.Data.SQLite.SQLiteConnection.Open()
   at SubSonic.SQLiteDataProvider.CreateConnection(String newConnectionString)
   at SubSonic.SQLiteDataProvider.CreateConnection()
   at SubSonic.SQLiteDataProvider.ExecuteScalar(QueryCommand qry)
   at SubSonic.DataService.ExecuteScalar(QueryCommand cmd)
   at SubSonic.ActiveRecord`1.Save(String userName)
   at SubSonic.ActiveRecord`1.Save()
   at (my line of code above)

and the inner exception is "Transaction Timeout"

I don't have any custom code in my generated DAL classes, or anything else clever that I can think of that would be causing this.

Anyone else encountered transaction problems like this or can someone suggest where I start looking for the problem?

thanks!

UPDATE: I notice mention of transaction-related things in the release notes for versions 1.0.61-65 (e.g. here), so perhaps updating SubSonic to work with the latest version of the .Net Data Provider would solve some of these issues...

Answer 1

We are using SQL lite to test are code related to database actions. SQL lite does not support nested Transaction. We had similar issues where we had NHibernate and .Net transaction. Ultimately we had to settle down using SQL Express to test the database related code.

Answer 2

In doing the revised sqlite provider for subsonic 2.x I created a full set of unit tests based on the existing subsonic sqlserver tests. (These tests were also checked in with the revised code.) The only tests that failed were the ones related to transactions (maybe the migration ones too). "The database file is locked" error message, as you have seen. Subsonic was written mainly for sql server that doesn't do file level locking like SQLIte does, so some things don't work; it would need to be rewritten to handle this better.

I have never used the TransactionScope as you have. I do my subsonic 2.2 transactions like this, and so far no problems with the SQLite provider. I can confirm that you need to use transactions with SQLite if dealing with multiple rows or it's really slow.

public void DeleteStuff(List<Stuff> piaRemoves)
{
    QueryCommandCollection qcc = new QueryCommandCollection();

    foreach(Stuff item in piaRemoves)
    {
        Query qry1 = new Query(Stuff.Schema);
        qry1.QueryType = QueryType.Delete;
        qry1.AddWhere(Stuff.Columns.ItemID, item.ItemID);
        qry1.AddWhere(Stuff.Columns.ColumnID, item.ColumnID);
        qry1.AddWhere(Stuff.Columns.ParentID, item.ParentID);
        QueryCommand cmd = qry1.BuildDeleteCommand();
        qcc.Add(cmd);
    }
    DataService.ExecuteTransaction(qcc);
}

Answer 3

I ended up using Paul's suggestion and rewrote my code to something like this:

    QueryCommandCollection qcc = new QueryCommandCollection();

    SomeDALObject x = new SomeDALObject()
    x.Property1 = "blah";
    x.Property2 = "blah blah";
    qcc.Add(x.GetSaveCommand());

    foreach (KeyValuePair<string, string> attribute in attributes)
    { 
        AnotherDALObject y = new AnotherDALObject()
        y.Property1 = attribute.Key
        y.Property2 = attribute.Value

        qcc.Add(y.GetSaveCommand());
    }

    DataService.ExecuteTransaction(qcc);

This is in fact much better since all preparation for the database hits are done before the transaction is opened, therefore the transaction will be open for much less time.

This won't work so well if you need to get back auto-generated IDs in order to run INSERTs for child records; you'll need to use a different approach for that.

I then hit some other threading/transaction problems: when I had multiple threads executing DataService.ExecuteTransaction() at the same time I would get AccessViolationExceptions and NullReferenceExceptions, basically a bit of a mess. But changing to use Paul's fork of SubSonic with the updated SQLDataProvider and also changing to use System.Data.SQLite v1.0.65.0 seems to instantly fixed it. Hooray!

UPDATE: Actually I'm still encountering threading problems using SubSonic with sqlite. Basically the SQLiteDataProvider in SubSonic isn't written to deal with multithreading. More to come...