I'm converting a forum from myBB to IPBoard (the conversion is done through a PHP script), however I have over 4 million posts that need to be converted, and it will take about 10 hours at the current rate. I basically have unlimited RAM and CPU, what I want to know is how can I speed this process up? Is there a way I can allocate a huge amount of memory to this one process? Thanks for any help!
A:
not sure how the conversion is done, but if you are importing a sql file , you could split it up to multiple files and import them at the same time. hope that helps :)
Aviatrix
2010-07-26 21:28:50
The conversion is done via a PHP script that converts 1500 posts at a time.
YCMaker
2010-07-26 21:31:52
A:
If you are saying that you have the file(s) already converted, you should look into MySQL Load Data In FIle for importing, given you have access to the MySQL Console. This will load data considerably faster than executing the SQL Statements via the source command.
If you do not have them in the files and you are doing them on the fly, then I would suggest having the conversion script write the data to a file (set the time limit to 0 to allow it to run) and then use that load data command to insert / update the data.
Brad F Jacobs
2010-07-26 21:32:26
Unfortunately, the conversion is done via a PHP script; basically, I want to know if there's a way to make the script run faster.
YCMaker
2010-07-26 21:34:29
Other than storing all the conversion into a file and then doing to load data, I do not think so. But posting the script, we may be able to help optimize it / make sure it is as optimum as it can be. But running multiple single queries on a MySQL DB is slow in general, being able to enter in via Extended Inserts or loading via a file tends to be much quicker.
Brad F Jacobs
2010-07-26 21:36:01
A:
You're not going to get a script to run any faster. By giving it more memory, you might be able to have it do more posts at one time, though. Change memory_limit in your php.ini file to change how much memory it can use.
You might be able to tell the script to do one forum at a time. Then you could run several copies of the script at once. This will be limited by how it talks to the database table and whether the script has been written to allow this -- it might do daft things like lock the target table or do an insanely long read on the source table. In any case, you would be unlikely to get more than three or four running at once without everything slowing down, anyway.
It might be possible to improve the script, but that would be several days' hard work learning the insides of both forums' database formats. Have you asked on the forums for IPBoard? Maybe someone there has experience at what you're trying to do.
staticsan
2010-07-27 00:14:32