views:

489

answers:

0

I have three tables. For simplicity let's say they are:

tbl1, tbl2, tbl3

both tbl2, and tbl3 have f-keys linking to tbl1

I am using 'load data infile...' as I really need the +16,000 writes/second I am getting right now with it.

I don't use auto-increment because I need the f-keys for my child records.

So I create my txtfile on the fly with my data and I have to include the f-keys for tbl2, and tbl3.

To obtain the current f-keys for the child table records I use 3 'select max(id) from tbl' on each table before writing the file and then just increment each counter for each record. The crappy thing is doing these 3 queries cost me all sorts of time and my speed dropped dramatically.

What are the ways around this? Create a hash of some sort?? A good answer must not cost any more time than what I'm already incurring as a result of using 'select max(id)'. Speed is the key here.

To clarify this is my pseudo code:

  tbl1.max = select max(id) from tbl...
  tbl2.max = select max(id) from tbl2
  tbl3.max = select max(id) from tbl3

  tbl1records.each do
    tbl1.file.write (id=tbl1.max, other stuff)
    tbl1.max++
  end

  tbl2records each do
    tbl2.file.write (fkey=tbl1.max, other stuff)
    tbl2.max++
  end

  tbl3records each do
    tbl3.file.write (fkey=tbl1.max, other stuff)
    tbl3.max++
  end

  load data infile tbl1.file
  load data infile tbl2.file
  load data infile tbl3.file

I AM just incrementing the counter from the previous selects. I only get the max from each table ONE TIME. (this is assuming we add 1000 or so to it so that it won't interfere with other inserts going on)

The problem is that I'm trying to write all 3 data files at once and then load them with three different load data infile calls. I HAVE to have a key of some sort to link the two child tables to the parent table. I can't use auto-increment cause I'm writing all of these files to disk and THEN uploading into the db