optimizing an sql query using inner join and order by

Question

I'm trying to optimize the following query without success. Any idea where it could be indexed to prevent the temporary table and the filesort?

EXPLAIN SELECT SQL_NO_CACHE `groups`.*
FROM `groups`
INNER JOIN `memberships` ON `groups`.id = `memberships`.group_id
WHERE ((`memberships`.user_id = 1) 
  AND (`memberships`.`status_code` = 1 AND `memberships`.`manager` = 0))
ORDER BY groups.created_at DESC LIMIT 5;`

+----+-------------+-------------+--------+--------------------------+---------+---------+---------------------------------------------+------+----------------------------------------------+
| id | select_type | table       | type   | possible_keys            | key     | key_len | ref                                         | rows | Extra                                        |
+----+-------------+-------------+--------+--------------------------+---------+---------+---------------------------------------------+------+----------------------------------------------+
|  1 | SIMPLE      | memberships | ref    | grp_usr,grp,usr,grp_mngr | usr     | 5       | const                                       |    5 | Using where; Using temporary; Using filesort | 
|  1 | SIMPLE      | groups      | eq_ref | PRIMARY                  | PRIMARY | 4       | sportspool_development.memberships.group_id |    1 |                                              | 
+----+-------------+-------------+--------+--------------------------+---------+---------+---------------------------------------------+------+----------------------------------------------+
2 rows in set (0.00 sec)


    +--------+------------+-----------------------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+
| Table  | Non_unique | Key_name                          | Seq_in_index | Column_name     | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+--------+------------+-----------------------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+
| groups |          0 | PRIMARY                           |            1 | id              | A         |           6 |     NULL | NULL   |      | BTREE      |         | 
| groups |          1 | index_groups_on_name              |            1 | name            | A         |           6 |     NULL | NULL   | YES  | BTREE      |         | 
| groups |          1 | index_groups_on_privacy_setting   |            1 | privacy_setting | A         |           6 |     NULL | NULL   | YES  | BTREE      |         | 
| groups |          1 | index_groups_on_created_at        |            1 | created_at      | A         |           6 |     NULL | NULL   | YES  | BTREE      |         | 
| groups |          1 | index_groups_on_id_and_created_at |            1 | id              | A         |           6 |     NULL | NULL   |      | BTREE      |         | 
| groups |          1 | index_groups_on_id_and_created_at |            2 | created_at      | A         |           6 |     NULL | NULL   | YES  | BTREE      |         | 
+--------+------------+-----------------------------------+--------------+-----------------+-----------+-------------+----------+--------+------+------------+---------+


     +-------------+------------+----------------------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table       | Non_unique | Key_name                                                 | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------------+------------+----------------------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| memberships |          0 | PRIMARY                                                  |            1 | id          | A         |           2 |     NULL | NULL   |      | BTREE      |         | 
| memberships |          0 | grp_usr                                                  |            1 | group_id    | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          0 | grp_usr                                                  |            2 | user_id     | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | grp                                                      |            1 | group_id    | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | usr                                                      |            1 | user_id     | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | grp_mngr                                                 |            1 | group_id    | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | grp_mngr                                                 |            2 | manager     | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | complex_index                                            |            1 | group_id    | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | complex_index                                            |            2 | user_id     | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | complex_index                                            |            3 | status_code | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | complex_index                                            |            4 | manager     | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | index_memberships_on_user_id_and_status_code_and_manager |            1 | user_id     | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | index_memberships_on_user_id_and_status_code_and_manager |            2 | status_code | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
| memberships |          1 | index_memberships_on_user_id_and_status_code_and_manager |            3 | manager     | A         |           2 |     NULL | NULL   | YES  | BTREE      |         | 
+-------------+------------+----------------------------------------------------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+

Answer 1

• Index on memberships column user_id (should already have one if it's PK)
• Index on memberships columns status_code and manager (both of them on same index)
• Index on groups column created_at (with default DESC if possible, I don't know if you can in mySql)

This is what I would do in a MS SQL Server but I guess same optimization can be used in mySql too.

Answer 2

Do you have all the "obvious" single-column indexes on the fields you join on, the fields in your where clause and the created_at field you order by?

Answer 3

The trouble is you need an index on groups in order to eliminate the filesort, but all of your where conditions are on memberships.

Try adding an index on groups on (id, created_at).

If that doesn't work, try tricking the optimizer like so using a subquery (keeping the aforementioned index on groups):

SELECT SQL_NO_CACHE `groups`.*
    FROM `groups`
    INNER JOIN (select group_id from `memberships`
       WHERE 
           `memberships`.user_id = 1
           AND `memberships`.`status_code` = 1
           AND `memberships`.`manager` = 0
    ) m on m.group_id=`groups`.id
    ORDER BY groups.created_at DESC LIMIT 5;

There should be an index on at least membershipships.user_id, but you could also gain some benefit from an index like (user_id, status, manager). I assume status and manager are flags that don't have a large range of possible values so it isn't that important as long as there's an index on user_id.

Answer 4

A (user_id, status_code, manager) (in any order) index on memberships would help.

Avoiding the sort would be difficult, because then you have to start the join in the groups table, which means you can't use all the (presumably pretty selective) where clauses that reference the memberships table until it is too late.

Answer 5

Thanks for posting details about the indexes you're using.

I've tested this, and tried omitting some of the indexes. The most index is memberships.complex_index which serves as a covering index. This allows the query to achieve its results by reading only the index; it doesn't have to read data rows for memberships at all.

None of the indexes on groups make any difference. It appears to use a filesort no matter what. Filesort in MySQL simply means it's doing a table-scan, which in some cases can be less costly than using an index. For instance, if every row of groups needs to be read to produce the result of the query anyway, why bother doing an unnecessary double-lookup by using an index? The optimizer can sense these cases and can appropriately refuse to use an index.

So aside from primary key indexes and the complex_index, I'd drop all the others, since they're not helping and can only add to the cost of maintaining these tables.