How to reduce the number of queries required to get this result.

Question

Hello,

I'm writing an application on top of CodeIgniter to better organize my ebook collection. I'm nearly done, but I realize my 'browse' page is running far too many queries - two per book - to get their information. Obviously not at all ideal, especially since I have about 1000 books to put into this system.

I currently have one model function which gets all of the books (will eventually be modified to take parameters - that's the next step) and another that gets the meta information for each returned book. The second function is the one which makes two queries for each book - one to get the information in the book table and another to get the tags associated with the book. Here are the two model functions:

Get the list of books:

function get_books() {
    $this->db->select('isbn')->order_by('title');
    $query = $this->db->get('books');
    $result = $query->result();
    return $result;
}

Get the book meta information:

function get_book_info($isbn) {
    // Grab the book from Amazon
    $amazon = $this->amazon->get_amazon_item($isbn);

    // Get the book info
    $this->db->select('title, publisher, date, thumb, filename, pages');
    $query = $this->db->get_where('books', array('isbn' => $isbn));
    $bookResult = $query->row();

    // Get the book's tags
    $this->db->select('tag');
    $this->db->from('tags AS t');
    $this->db->join('books_tags AS bt', 'bt.tag_id = t.id', 'left');
    $this->db->where('bt.book_id', $isbn);
    $this->db->order_by('t.tag');
    $tagQuery = $this->db->get();
    foreach ($tagQuery->result() as $row) {
        $tagResult[] = $row->tag;
    }
    $tagResult = implode(', ', $tagResult);

    // Send data
    $data = array(
        'isbn' => $isbn,
        'thumb' => $bookResult->thumb,
        'title' => strip_slashes($bookResult->title),
        'file' => $bookResult->filename,
        'publisher' => strip_slashes($bookResult->publisher),
        'date' => date('F j, Y', strtotime($bookResult->date)),
        'pages' => $bookResult->pages,
        'tags' => $tagResult,
        'rating' => $amazon->Items->Item->CustomerReviews->AverageRating,
        'raters' => $amazon->Items->Item->CustomerReviews->TotalReviews
    );
    return $data;
}

I'm certain there's a way to write one or two queries that will gather all the records into objects I can then filter through, rather than having to write two queries for each one, but I have no idea where to even start trying to write that. Any suggestions are welcome.

Thanks much, Marcus

Answer 1

If I got you right: in the table books there is all the data about the books: so doing this:

$this->db->select('*')->order_by('title');
$query = $this->db->get('books');
$result = $query->result();
return $result;

should return you all the data about your books and you shouldn't need to cycle again to get data.

Answer 2

I'm not familiar with CodeIgniter at all, but I think there are some general practices you can incorporate.

• If this is a browse page - isn't there pagination? Paginating the results should drastically cut down the number of queries you have to run per page load.
• Have one function (say, get_books_info()) you call that retrieves all the tags & meta info for all the books returned by your get_books() function. Then reference that array from your get_book_info(). You can even trigger get_books_info() from get_book_info() - so you only need to do the work if you need the data. Kind of lazy loading I think.

Answer 3

What you want to do is:

1. Grab a sorted list of all of your books and their tags,
2. Style them out as HTML, with tags and ratings.

Grab your books and tags together, have a variable to keep track of the last ISBN you wrote out, and only build up your entry when the ISBN changes. So, pull a set like this:

Book   | Tag
------ | ----------------
Book A | Fiction
Book A | Fantasy
Book B | Mystery
Book C | Science Fiction

Then, write out the "basic book info" for every time the book changes within your loop. Obviously, you'll want more fields than just Book and Tag (e.g., ISBN).

If your Amazon information comes from Amazon, you're probably going to have no choice about making repetitive calls to their API (unless they have a "batch" mode or something, wherein you could submit an array of ISBN's?).

Answer 4

With some help from this topic and in others in creating a better query, I was able to resolve this with the following code:

function get_book_info() {

    /*
     * SELECT b.isbn, b.title, b.publisher, b.date, b.thumb, b.filename, b.pages, t.tag
     * FROM books AS b
     * INNER JOIN books_tags AS bt ON b.isbn = bt.book_id
     * INNER JOIN tags AS t ON bt.tag_id = t.id
     * ORDER BY b.title, t.tag
     */

    $this->db->select('b.isbn, b.title, b.publisher, b.date, b.thumb, b.filename, b.pages, t.tag');
    $this->db->from('books AS b');
    $this->db->join('books_tags AS bt', 'b.isbn = bt.book_id', 'inner');
    $this->db->join('tags AS t', 'bt.tag_id = t.id', 'inner');
    $this->db->order_by('b.title, t.tag');
    $query = $this->db->get();
    $result = $query->result();

    $counter = '';
    $record = $meta = $tags = array();
    $count = count($result);
    $i = 1;

    foreach ($result as $book) {
        // If this is not the last row
        if ($i < $count) {
            // If this is the first appearance of this book
            if ($counter != $book->isbn) {
                // If the meta array already exists
                if ($meta) {
                    // Add the combined tag string to the meta array
                    $meta['tags'] = implode(', ', $tags);
                    // Add the meta array
                    $record[] = $meta;
                    // Empty the tags array
                    $tags = array();
                }
                // Reset the counter
                $counter = $book->isbn;
                // Grab the book from Amazon
                $amazon = $this->amazon->get_amazon_item($book->isbn);
                // Collect the book information
                $meta = array(
                    'isbn' => $book->isbn,
                    'title' => strip_slashes($book->title),
                    'publisher' => strip_slashes($book->publisher),
                    'date' => date('F j, Y', strtotime($book->date)),
                    'thumb' => $book->thumb,
                    'file' => $book->filename,
                    'pages' => $book->pages,
                    'rating' => $amazon->Items->Item->CustomerReviews->AverageRating,
                    'raters' => $amazon->Items->Item->CustomerReviews->TotalReviews
                );
                // Add the tag to the tags array
                $tags[] = $book->tag;
            } else {
                // All we need is the tag
                $tags[] = $book->tag;
            }
        // If this is the last row
        } else {
            // If this is the first appearance of this book
            if ($counter != $book->isbn) {
                // Grab the book from Amazon
                $amazon = $this->amazon->get_amazon_item($book->isbn);
                // Collect the book information
                $meta = array(
                    'isbn' => $book->isbn,
                    'title' => strip_slashes($book->title),
                    'publisher' => strip_slashes($book->publisher),
                    'date' => date('F j, Y', strtotime($book->date)),
                    'thumb' => $book->thumb,
                    'file' => $book->filename,
                    'pages' => $book->pages,
                    'rating' => $amazon->Items->Item->CustomerReviews->AverageRating,
                    'raters' => $amazon->Items->Item->CustomerReviews->TotalReviews
                );
                // Add the tag to the tags array
                $tags[] = $book->tag;
                // Add the combined tag string to the meta array
                $meta['tags'] = implode(', ', $tags);
                // Add the meta array
                $record[] = $meta;
            } else {
                // All we need is the tag
                $tags[] = $book->tag;
                // Add the combined tag string to the meta array
                $meta['tags'] = implode(', ', $tags);
                // Add the meta array
                $record[] = $meta;
            }
        }
        $i++;
    }

    return $record;
}

There may very well be a better way to handle this, but this was how my logic saw it. And only one query, total.