There are quite a few independent and not-so-independent studies comparing traditional RDBMSs but I haven't managed to find any good material on in-memory databases. I am primarily interested in ones specialized for OLTP.
So far, I managed to find generic white papers on TimesTen and MySQL Cluster, but I have yet to see a head-to-head comparison. There are other alternatives (e.g. from IBM), but there's even less easily available material.
This wiki article gives a comparison of most of the well known RDBMS's: Comparison of relational database management systems
No comparison, but lists the in-memory RDBMS's: In-memory database
Also, it is going to depend on your intended use to some extent. What are your constraints and minimum performance requirements?
I think your best option is to obtain trial (or free) versions and set up your own benchmarks.
The information is scattered all over the web, but here's what I found out:
Introduction to database benchmarking
The first thing that you need to do is figure out a standard benchmark to use for comparing your in-memory database choices. A wiki page on Oracle gives an overview of the industry standard database benchmarks. If you can find results for the same industry standard benchmarks for TimesTen and MySql Cluster, you'll be able to compare the two. Finding a detailed comparison of all the in-memory databases is quite rare.
Summary: Since you mentioned OLTP, the industry standard benchmark that you're probably interested in is TPC-E which simulates the OLTP workload of a brokerage firm. Other industry standard benchmarks measure decision support and e-commerce database transactions.
Open source database benchmark kits
The Open Source Development Labs Database Test Suite is the most comprehensive open source implementation of those industry standard benchmarks I mentioned earlier. They have four main suite of tests numbered DBT1, DBT2, DBT3, DBT4. They haven't implemented the TPC-E benchmark, but the Database Test 2 (DBT2) is what you are looking for since it simulates OLTP transactions. Everyone simply refers to running the DBT2 benchmark instead of spelling it out as OSDLB DBT2.
Results
MySql
You'll need to register to get the white paper emailed to you, but here is the summary that I took from their site:
Ultimately, MySQL Cluster was able to reach 100,000 transactions per minute in an 8-node configuration. This showed impressive scaling improvements when compared to a 2-node cluster which was able to obtain 26,000 transactions per minute. These performance improvements were realized through the use of new Multi Core Intel Xeon servers based on the Intel Core T micro architecture.
Oracle
Unfortunately the DBT2 does not support TimesTen. However I found a whitepaper that gives detailed benchmarking information for TimesTen in an OLTP workload.
IBM
Other Resources
Almost every DBMS will do the same thing but differently, therefore you should look for the fastest DBMS you can find, if what you need is speed then you should use TimesTen wich is accepted as one of the fastest solutions, the thing with open-source DBMS is that, some of the more advanced features (such as replication) are nowhere near what you'd find on commercial alternatives. Quite simply, most users don't need replication at the levels that Oracle, DB2, or MS-SQL offer; therefore, PostgreSQL and MySQL developers don't get feel the need to improve it.
This answer was obtained after looking for the fastest between RDBMS mentioned here and some already known by me, I went 2 by 2 looking for the best and then looking for the best between the winner and the other, and the result was that TimesTen is the fastest between:
So, you should use for the application you are mentioning and the conditions set TimesTen.
If you are looking for open source main memory database alternative, you can check out CSQL.
You should check this post TimesTen and solidDB in-memory databases comparison There is a nice comparison of TimesTen and solidDB.
VoltDB is an in-memory highly-scalable OLTP database that provides SQL/ACID and runs on commodity hardware in a shared-nothing cluster. It is open-source and exists in both a free/community (GLP3) version as well as a commercially supported version.
There is a post in the VoltDB forums discussing some of the other in-memory OLTP RDBMS solutions at http://community.voltdb.com/node/95. Feel free to review and add to that posting.