Recording user data for heatmap with JavaScript

Question

Hi,

I was wondering how sites such as crazyegg.com store user click data during a session. Obviously there is some underlying script which is storing each clicks data, but how is that data then populated into a database? It seems to me the simple solution would be to send data via AJAX but when you consider that it's almost impossible to get a cross browser page unload function setup, I'm wondering if there is perhaps some other more advanced way of getting metric data.

I even saw a site which records each mouse movement and I am guessing they are definitely not sending that data to a database on each mouse move event.

So, in a nutshell, what kind of technology would I need in order to monitor user activity on my site and then store this information in order to create metric data? I am not looking to recreate GA, I'm just very interested to know how this sort of thing is done.

Thanks in advance

Answer 1

If you're just looking for interaction, you could replace your <input type="button"> with <input type="image">. These are automatically submitted with X,Y coordinates of where the user has clicked.

jQuery also has a good implementation of the mousemove even binding that can track the current mouse position. I don't know your desired end result, but you could setTimeOut(submitMousePosition, 1000) to send an ajax call with the mouse position every second or something like that.

Answer 2

The fundamental idea used by many tracking systems uses a 1x1px image which is requested with extra GET parameters. The request is added to server log file, then log files are processed to generate some statistics. So a minimalist click tracking function might look like this:

document.onclick = function(e){
  var trackImg = new Image();
  trackImg.src = 'http://tracking.server/img.gif?x='+e.clientX+'&y='+e.clientY;
}

AJAX wouldn't be useful because it is subject to same-origin policy (you won't be able to send requests to your tracking server). And you'd have to add AJAX code to your tracking script. If you want to send more data (like cursor movements) you'd store the coordinates in a variable and periodically poll for a new image with updated path in the GET parameter.

Now there are many many problems:

• cross-browser compatibility - to make the above function work in all browsers that matter at the moment you'd probably have to add 20 more lines of code
• getting useful data
  • many pages are fixed-width, centered, so raw X and Y coordinates won't let you create visual overlay of clicks n the page
  • some pages have liquid-width elements, or use a combination of min- and max-height
  • users may use different font sizes
  • dynamic elements that appear on the page in response to user's actions
• etc. etc.

When you have the tracking script worked out you only need to create a tool that takes raw server logs and turns them into shiny heatmaps :)

Answer 3

I really don't see why do you think that is impossible to store all click points in one user session to the database?

Their moto is "See Where People Click" Once when you gather enough data it is fairly easy to make heat maps in batch processes.

People are really underestimating databases, indexing and sharding. The only hard thing here is to gather enough money for underlying architecture :)