Counting Unique Users using Mapreduce for Java Appengine

Question

I'm trying to count the number of unique users per day on my java appengine app. I have decided to use the mapreduce framework (mapreduce.appspot.com) for java appengine to do this calculation offline. I've managed to create a map reduce job that goes through all of my entities which represent a single users session event. I can use a simple counter as well. I have several questions though:

1) How do I only increment a counter once for each user id? I am currently mapping over entities which contain a user id property but many of these entities may contain the same user id so how do I only count it once?

2) Once I have these results of the job stored in these counters - how can I persist them to the datastore? I see the results of the counters on the mapreduce's status page but I want these results automatically persisted to the datastore.

Ideas?

Answer 1

Why on Earth are you using map-reduce for this? Maybe I'm missing something, but this seems like a non-parallelizable problem, so there's no good reason for using a parallel-computing framework.

Why not just do something like this:

SELECT COUNT(DISTINCT user_id_column) FROM user_session_table 

See: http://www.w3schools.com/sql/sql_func_count.asp

Answer 2

I haven't actually used the MapReduce functionality yet, but my theoretical understanding is that you can write things to the datastore from within your mapper. You could create an Entity type called something like UniqueCount, and insert one entity every time your mapper sees an ID that it hasn't seen before. then you can count how many unique ID's you have. In fact, you can just update a counter every time you find a new unique entity. You may want to google "sharded counter" for hints on creating a counter in the datastore that can handle high throughput.

Eventually, when they finish the Reduce functionality, I imagine this whole task will become pretty trivial.