Optimizing a MySQL query to avoid "Using temporary" and "Using filesort"

Question

I know there's a thousand similar questions out there, but none deal with as convoluted a query as mine (and my MySQL skills aren't to the point to really understand how to adapt them.)

Here it is:

explain select
  `ev`.`EventID` AS `EventID`
  ,`ev`.`EventName` AS `EventName`
  ,concat(`ev`.`EventDate`,' ',`ev`.`StartTime`) AS `EventDT`
  ,`ev`.`NumberTicketsAvailable` AS `TotalTickets`
  ,`ev`.`Soldout` AS `Soldout`
  ,count((case when (`ec`.`CartStatus` = 'InCart') then 1 else NULL end)) AS `InCartCount`
  ,count((case when (`ec`.`CartStatus` = 'InPayment') then 1 else NULL end)) AS `InPaymentCount`
  ,count((case when (`ec`.`CartStatus` = 'Paid') then 1 else NULL end)) AS `PaidCount`
  ,count((case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 604800 second) > now())) then 1 else NULL end)) AS `PaidOverWeek`
  ,count((case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 432000 second) > now())) then 1 else NULL end)) AS `PaidOverFiveDays`
  ,count((case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 86400 second) > now())) then 1 else NULL end)) AS `PaidOverDay`
  ,count((case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 43200 second) > now())) then 1 else NULL end)) AS `PaidOverHalfDay`
  ,count((case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 21600 second) > now())) then 1 else NULL end)) AS `PaidOverQuarterDay`
  ,count((case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 10800 second) > now())) then 1 else NULL end)) AS `PaidOverThreeHours`
  ,count((case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 3600 second) > now())) then 1 else NULL end)) AS `PaidOverHour`
from (`Events` `ev`
  left join (`Events_EventCart_Rel` `eecr`
    left join `EventCart` `ec`
      on((`eecr`.`EventCartID` = `ec`.`EventCartID`)))
    on((`ev`.`EventID` = `eecr`.`EventID`)))
where (`eecr`.`Active` = 1 AND `eecr`.`Deleted` = 'No')
group by
  `ev`.`EventID`
  ,`ev`.`EventName`
  ,`ev`.`EventDate`
  ,`ev`.`StartTime`
  ,`ev`.`NumberTicketsAvailable`
  ,`ev`.`Soldout`;

The results of this look like this:

+-id-+-select_type-+-table-+--type--+--------possible_keys--------+----key----+-key_len-+----------ref----------+--rows--+---------------------------Extra---------------------------+
|   1| SIMPLE      | eecr  | index  | EventID,EventID_2,EventID_3 | EventID_3 | 10      | {null}                | 17609  | Using where; Using index; Using temporary; Using filesort |
|   1| SIMPLE      | ev    | eq_ref | PRIMARY                     | PRIMARY   | 4       | eecr.EventID          | 1      | Using where                                               |
|   1| SIMPLE      | ec    | eq_ref | PRIMARY                     | PRIMARY   | 4       | eecr.EventCartID      | 1      |                                                           |
+----+-------------+-------+--------+-----------------------------+-----------+---------+-----------------------+--------+-----------------------------------------------------------+

And the table definitions:

CREATE TABLE IF NOT EXISTS `Events` (
  `EventID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `EventName` varchar(150) NOT NULL,
  `StartTime` char(8) NOT NULL DEFAULT '00:00:00',
  `EndTime` char(8) NOT NULL DEFAULT '00:00:00',
  `EventDate` varchar(20) NOT NULL,
  `NumberTicketsAvailable` smallint(6) DEFAULT NULL,
  `Soldout` enum('yes','no') DEFAULT 'no',
  #...
  PRIMARY KEY (`EventID`),
  KEY `EndTime` (`EndTime`,`EventDate`),
  KEY `StartTime` (`StartTime`,`EventDate`),
  KEY `EventDate` (`EventDate`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1

CREATE TABLE IF NOT EXISTS `Events_EventCart_Rel` (
  `ID` int(11) NOT NULL AUTO_INCREMENT,
  `EventCartID` int(11) NOT NULL,
  `EventID` int(11) NOT NULL,
  `DateAdded` datetime NOT NULL,
  `PersonID` int(11) NOT NULL,
  `SeatTypeID` int(11) NOT NULL,
  `MealChoiceID` int(11) NOT NULL,
  `Active` tinyint(1) NOT NULL DEFAULT '1',
  `Deleted` enum('Yes','No') NOT NULL DEFAULT 'No',
  `ModifiedByAdmin` enum('Yes','No') NOT NULL DEFAULT 'No',
  PRIMARY KEY (`ID`),
  KEY `EventID` (`EventID`,`PersonID`),
  KEY `EventCartID` (`EventCartID`),
  KEY `EventID_2` (`EventID`),
  KEY `EventID_3` (`EventID`,`EventCartID`,`Active`,`Deleted`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1

CREATE TABLE IF NOT EXISTS `EventCart` (
  `EventCartID` int(11) NOT NULL AUTO_INCREMENT,
  `RegistrantsID` int(11) NOT NULL DEFAULT '0',
  `DateRecordCreated` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `DateRecordModified` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
  `CartStatus` enum('InCart','InPayment','Paid') NOT NULL DEFAULT 'InCart',
  `ModifiedByAdmin` enum('yes','no') NOT NULL DEFAULT 'no',
  PRIMARY KEY (`EventCartID`),
  KEY `rid` (`RegistrantsID`)
) ENGINE=InnoDB  DEFAULT CHARSET=latin1

To preempt any puzzled looks at a few of those columns - yes, there are quite a few legacy-type things in there that I didn't have time to fix in code.

Answer 1

I've found that in MySQL at least, almost any query using GROUP BY invokes a temporary table. This is where your big performance cost goes. Try examining where it's spending its time using the profiler:

edit: I'm correcting the following to SET PROFILING (not SET PROFILES):

SET PROFILING = On;
SELECT ...the whole query you want to profile...
SHOW PROFILES;
SHOW PROFILE FOR QUERY 1;

See http://dev.mysql.com/doc/refman/5.1/en/show-profiles.html for more details.

There's not much you can do to fix this. It's sometimes preferable for performance reasons to eliminate the GROUP BY and aggregate functions:

select
  `ev`.`EventID` AS `EventID`
  ,`ev`.`EventName` AS `EventName`
  ,concat(`ev`.`EventDate`,' ',`ev`.`StartTime`) AS `EventDT`
  ,`ev`.`NumberTicketsAvailable` AS `TotalTickets`
  ,`ev`.`Soldout` AS `Soldout`
  ,case when (`ec`.`CartStatus` = 'InCart') then 1 else 0 end AS `InCartCounter`
  ,case when (`ec`.`CartStatus` = 'InPayment') then 1 else 0 end AS `InPaymentCounter`
  ,case when (`ec`.`CartStatus` = 'Paid') then 1 else 0 end AS `PaidCounter`
  ,case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 604800 second) > now())) then 1 else 0 end AS `PaidOverWeekCounter`
  ,case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 432000 second) > now())) then 1 else 0 end AS `PaidOverFiveDaysCounter`
  ,case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 86400 second) > now())) then 1 else 0 end AS `PaidOverDayCounter`
  ,case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 43200 second) > now())) then 1 else 0 end AS `PaidOverHalfDayCounter`
  ,case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 21600 second) > now())) then 1 else 0 end AS `PaidOverQuarterDayCounter`
  ,case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 10800 second) > now())) then 1 else 0 end AS `PaidOverThreeHoursCounter`
  ,case when ((`ec`.`CartStatus` = 'Paid') and ((`ec`.`DateRecordModified` + interval 3600 second) > now())) then 1 else 0 end AS `PaidOverHourCounter`
from `Events` `ev`
inner join `Events_EventCart_Rel` `eecr`
  on `ev`.`EventID` = `eecr`.`EventID` 
inner join `EventCart` `ec`
   on `eecr`.`EventCartID` = `ec`.`EventCartID`
where `eecr`.`Active` = 1 and `eecr`.`Deleted` = 'No'

Then in your application code, fetch all the rows, and loop over them, calculating the aggregate counts as you go. For instance in PHP:

$stmt = $pdo->query($sql);
$events = array();
$counters = array("InCartCounter", "InPaymentCounter", "PaidCounter",
  "PaidOverWeekCounter", "PaidOverFiveDaysCounter", "PaidOverDayCounter",
  "PaidOverHalfDayCounter", "PaidOverQuarterDayCounter", 
  "PaidOverThreeHoursCounter", "PaidOverHourCounter");

while ($row = $stmt->fetch())
{
  if (!isset($events[$row["EventID"]])) {
    $events[$row["EventID"]] = $row;
  } else {
    foreach ($counters as $key) {
      $events[$row["EventID"]][$key] += $row[$key];
    }
  }
}

It looks like a lot of code and trouble to do something that SQL should be able to do more efficiently, but in the case of MySQL and GROUP BY writing more application code is often better.

PS: In the example SQL query, I changed your joins to inner joins. I don't think you need outer joins.