Output buffering and large MySQL result sets in PHP 5

Question

Hey guys

I'm trying to build an XML feed from a database with a ginormous table, almost 4k records. I want to use output buffering to get it to spit out the XML but the script still keeps on timing out.

ob_start();
$what = 'j.*, (select description from tb_job_type as jt WHERE jt.jobtype_id =  j.job_type_id) as job_type,';
$what .= '(select description from tb_location as l WHERE l.location_id = j.location_id) as location,';
$what .= '(select description from tb_industry as i WHERE i.industry_id = j.industry_id) as industry';
$where = ('' != $SelectedType) ?  'j.job_ad_type="' . $SelectedType .'"' : '';
$process = $db->executeQuery('SELECT ' . $what . ' FROM tb_job_ad as j' . $where);

while($result = mysql_fetch_array($process))
{
    $result['job_title_url']        = $form->urlString($result['job_title']);
    $result['job_title']            = htmlspecialchars($result['job_title'], ENT_QUOTES, 'UTF-8');
    $result['short_description']    = htmlspecialchars($result['short_description'], ENT_QUOTES, 'UTF-8');
    $result['full_description']     = htmlspecialchars($result['full_description'], ENT_QUOTES, 'UTF-8');
    $result['company_name']         = ucwords(strtolower($result['company_name']));
    $tpl->assignToBlock('ITEMS', $result);
}
$cheese = ob_get_contents();    
$actualize = $tpl->actualize('FEED');
header("Expires: Mon, 26 Jul 1997 05:00:00 GMT" );
header("Last-Modified: " . gmdate( "D, d M Y H:i:s" ) . "GMT" );
header("Cache-Control: no-cache, must-revalidate" );
header("Pragma: no-cache" );
header("Content-type: text/xml");
echo $actualize;
ob_flush();
print $cheese;
ob_end_clean();

This seems to be the line that makes the script choke:

$tpl->assignToBlock('ITEMS', $result);

Help please?

Thanks

Midiane.

Answer 1

Could it be that you have a rather slow query?
Compare the output of

set_time_limit(60);
$process = $db->executeQuery('EXPLAIN SELECT ' . $what . ' FROM tb_job_ad as j' . $where);
while($result = mysql_fetch_array($process, MYSQL_ASSOC)) {
  echo join(' | ', $result), "<br />\n";
}

with Optimizing Queries with EXPLAIN.

Answer 2

You can use set_time_limit(0) to allow your script to run forever without any timeout and wait until it completes execution.

Answer 3

The timeout is almost certainly happening because your query is slow -- and you could almost certainly improve it's performance by making sure you've got the right columns indexed, and doing some JOINs.

What if you rewrote you query construction like this:

$q = 'SELECT 
  j.*, 
  jt.description as job_type, 
  l.description as location, 
  i.description as industry
FROM tb_jobs AS j
INNER JOIN tb_job_type AS jt ON j.job_type_id = jt.jobtype_id 
INNER JOIN tb_location AS l ON j.location_id = l.location_id
INNER JOIN tb_industry AS i ON j.indsutry_id = i.industry_id';

if (! empty($SelectedType)){
  $where = "WHERE j.job_ad_type = '$SelectedType'";
}else{
  $where = '';
}

$q .= $where;

Make sure that all your foreign keys columns (j.location_id, etc) are indexed.

If you want output to start sooner, you'll want to - Query the database - Output all your headers, etc. - write you while loop like:

ob_end_flush();flush();
while($row = mysql_fetch_assoc($process)
{
  ob_start();
  //build your ITEM and echo it here
  ob_end_flush(); flush();
}

PS: I also noticed another SQL-Related question of yours here on SO. I think you'd be very well served by gaining a better understanding of how SQL works. I highly recommend getting a copy of "SLQ Clearly Explained" -- it delivers exactly what the title promises -- a clear explanation of SQL (in general, without getting bogged down discussing various implementations)