python sqlalchemy performance?

views:

769

answers:

+1 Q:

python sqlalchemy performance?

HI ， I made a ICAPServer (similar with httpserver) for which the performance is very important. The DB module is sqlalchemy. I then made a test about the performance of sqlalchemy, as a result, i found that it takes about 30ms for sqlalchemy to write <50kb data to DB (Oracle), i don`t know if the result is normal, or i did something wrong? BUT, no matter right or wrong, it seems the bottle-neck comes from the DB part. HOW can i improve the performance of sqlalchemy? OR it is up to DBA to improve Oracle?

BTW, ICAPServer and Oracle are on the same pc , and i used the essential way of sqlalchemy..

+1 A:

You can only push SQLAlchemy so far as a programmer. I would agree with you that the rest of the performance is up to your DBA, including creating proper indexes on tables, etc.

B.R. 2009-07-10 17:28:23

But is "30ms for sqlalchemy to write <50kb data to DB (Oracle)" normal ? Is there any track record about performacne of sqlalchemy?BTW, Proper indexes things were done already.

Andy 2009-07-10 17:40:12

I'm not sure about what a normal baseline for SQLAlchemy's performance is, but 50 kb in 30ms seems pretty fast--I've had longer round-trip times to make a single query than that. I would have to agree with Luper's comment below that you should probably profile to see where the slowdown is (in retrospect, I'm a bit shamed that I didn't think to mention that).

B.R. 2009-07-10 17:46:43

+4 A:

You should first measure where your bottleneck is, for example using the profile module.

Then optimize, if you have the possibility to, the slowest part of the system.

Luper Rouch 2009-07-10 17:41:18

thanks a lot, all of you (i don`t know how to add comments for all of you ,so choose Luper)I think the slowest part must be DB-i/o, and i`d like to describe what my ICAPAerver is doing , as follows:There r two main steps and two thread;1st step, ICAPServer receives data from clients, put the data in a queue (50kb <1ms)2nd step, another thread pop data from the queue, and write them to DBSO, if 2nd step is too slow, the queue will fill up memory with those data...

Andy 2009-07-10 18:33:02

+1 A:

I had some issues with sqlalchemy's performance as well - I think you should first figure out in which ways you are using it ... they recommend that for big data sets is better to use the sql expression language. Either ways try and optimize the sqlalchemy code and have the Oracle database optimized as well, so you can better figure out what's wrong. Also, do some tests on the database.

hyperboreean 2009-07-10 17:46:49

i am using the ORM part , and data not so big (<50kb per commit())

Andy 2009-07-10 18:38:30

i tried printing time stamp before and after "commit()", take the time difference, results r about 30ms

Andy 2009-07-10 18:43:08

I forgot to ask one more thing: is your database locally or are you accessing it over a network. This might induce some latency too.

hyperboreean 2009-07-10 19:00:03

The DB is running locally

Andy 2009-07-10 19:05:33

I think the slowest part must be DB-i/o, and i`d like to describe what my ICAPServer is doing , as follows: There r two main steps and two thread; 1st step, ICAPServer receives data from clients, put the data in a queue (50kb <1ms); 2nd step, another thread pop data from the queue, and write them to DB. So, if 2nd step is too slow, the queue will fill up memory with those data. So, i have to speed up the DB part...I use "print time stamp" way on every important part, and the DB i/o always the slowest

Andy 2009-07-10 19:08:56

Wondering if you have any suggestion about the queue problem...

Andy 2009-07-10 19:12:27

ansaurus

tags:

views:

answers:

python sqlalchemy performance?

related questions