Is there a way to grab files from other servers and dump them into the DB asynchronous?

Question

Hello guys,

I need to create a php script that takes lots of URL's via POST and then loads the corresponding files and dumps them in the DB. The thing is that I would like to do it asynchronous, so that if I have 1000 files to get, the script won't hang till all the files are loaded. Also, every time a file it's done loading, I need to know so that I can insert it in the DB

Any ideas are appreciated.

Answer 1

Split the script in two parts - first to collect the URLs and the second is a shell script to be run from background to get the urls inserted in the database and fetch them.

So basically the process is as follows:

Script1:

• Gets POST
• Inserts into database
• Call script 2 with
• shell_exec to run in background

Script2:

• Get all the urls from urls_to_download
• Fetch the URLS (consequentially or parallel, depends on you)
• Do stuff with them
• Save them to database.

And you are done. The POST in script1 returns immediately and the script2 is then running. All left for you is to check status (poll from database through AJAX may be) for the URLS if you want to show some information about progress.

Answer 2

PHP is not multithreaded and perfectly synchronous. So you may not do this using PHP alone.

But you may use another language to do this task, for example JavaScript (which is asynchronous). Try node.js. It is lightning fast and has mysql bindings ;) Use http.Client to make the requests to the sites.