SQLite non-exclusive RESERVED lock?

Question

I've been looking into improving SQLite performance for my site, especially with regard to transactions. In essence what I'm looking for is a way to defer database writes in a process so that they can all be done at once. However, while I'm accumulating update queries, I would like other processes to be able to both read from and write to the database, and only lock the file for writing once a commit is issued in a process.

In looking at the documentation, it seems as though once an update command is issued in a transaction the process gets a RESERVED lock, which (if I remember correctly) means that any other process that attempts to either add an update query to its own transaction or commit the transaction is unable to do so, and therefore blocks until the transaction commits on the process with the lock.

I'm sure there are very good data integrity reasons against this particular feature. All I can say is that in my case there's no danger in performing these updates simultaneously.

One solution is that in each process I could accumulate the text of the queries I wish to invoke in an array, then loop down it once I'm ready to write, but I'm wondering if it's possible the SQLite transaction can be made to do this for me automatically.

update: What I mean when I say "doing all my updates at once" is essentially using transactions in SQLite to only get an EXCLUSIVE lock and write to disk once per processes, rather than once per query. This leads to 100x speedups using SQLite.

I've done some basic testing and it seems that once you have multiple processes adding queries to their transactions, once you hit an update query that process attempts to get the RESERVED lock. Since only once process can have the reserved lock, this means that any other processes attempting to get the lock will block until the process with the lock finished the transaction.

I will admit that this problem might be a premature optimization as I have yet to encounter any performace penalties, but I have run some simple tests and 100 users each creating and running a transaction with 100 queries takes about 4 seconds in PHP on my machine.

Answer 1

Surely trying to do all your writes at once is going to lead to more contention, not less? If you are keeping your writes short, then transctions should be short as well, and blocking will be minimal. Could you supply more specific information please.

Many blocking problems are due to not having the appropriate indexes.

Answer 2

SQLite supports ATTACH to attach one database to another database. Perhaps you can accumulate your data in a separate database, and when you are ready to merge the accumulated rows, attach the separate database, copy the rows in a single statement, and detach.

Edit: a similar suggestion to the OP was made on a mailing list thread at sqlite-users with some follow-up discussion.