Using the log to generate analytics?

Question

Hi,

I'm using app engine java. I'm interested in collecting some analytics, but this is difficult since I need to make datastore classes for all the info I'm interested in tracking - it's going to detract from our 100 index quota, and just introduce extra complexity to my project.

I was reading through the dev forum, an app engine project member suggested just using the log to record data, then download all the logs for a given day, and parse it locally to do our analytics. I wonder if anyone else has experience with this? I'd be interested in doing something like:

import java.util.logging.Logger;

public class MyServlet extends HttpServlet {

    public void doGet(HttpServletRequest req, HttpServletResponse resp) {
        log.info("Serving page '/example/farm'");

        Farm farm = null;
        try { 
            farm = datastore.getFarm();
            log.info("Got farm ok.");
        } catch (Exception ex) {
            log.info("No farm found!");
        }                                    
    }
}

if I downloaded the logs each day, I could parse it and count the # of times the "Serving page..." message appears, and get an idea for how many times that page is served. Same for the success or fail rate of finding the requested Farm from the other message. Is this sensible? I'm not sure how slow it is to use the log, or if I'm better off trying to store these stats in the datastore manually.

Another concern is if this page gets hit thousands of times a day, the log file may be enormous and impractical to download etc?

Thank you

Answer 1

There are third party services available for the purpose. You can use them in your application. I can't suggest you to do it as your own. Check out Google Analytics

Answer 2

I've done this sort of internal analytics before and I've always spawned events on the task queue to record the information in the datastore. It works well because you can build the custom analysis tools inside the app and make them accessible from anywhere via the browser.

The 100 limit on the indexes can be overcome easily enough by switching to a paid account. You may find that you'll stay within the free quotas and not actually pay anything, but it will give you more indexes.

BTW, the Google Analytics strategy is an interesting one. You could simply use urlfetch() or create task queue events on various admin-only URLs. Then use analytics to see how many times those pages were touched.