tags:

views:

331

answers:

1
+2  Q: 

How do web analytics platforms (GA for example) calculate unique visitors?

Counting distinct visitors is not an easy task. In web analytics for example, a visitor can visit on Monday and Thursday, but when counting unique visitors over that week, I'd expect to count that visitor only once.

count (distinct userid) for 10M visits over a month can't run too fast as aggregations can't be applied (because count distinct is not an "addable" measure).

My question is: How do Google Analytics and other web analytics platforms return unique visitors so fast? I assume statistical estimations are used. What kind? How?

A: 

They set a cookie with a reasonable expiration. If you have the cookie already, you've come back.

great_llama  2009-05-09 13:10:24
the question is not specifically about cookies, it is more general - how to estimate count distinct for a high cardinality column ?
 2009-05-10 07:44:35
"How do Google Analytics and other web analytics platforms return unique visitors so fast?" They don't count distinct.
great_llama  2009-05-10 12:24:17
... rather, they don't do it at reporting time.
great_llama  2009-05-10 12:25:14
I assume they have a batch operation that pre-calculate everything needed for the report to run fast. What is that pre-calculate operation?
 2009-05-10 12:31:23

related questions

How do I print unique elements in Perl array?
Identifying unique hits with javascript
What can C++ do that is too hard or messy in any other language?
Ideas on making a javascript object name unique in ASP.Net?
Should I check for DB constraints in code or should I catch exceptions thrown by DB
Normalizing a list of items in MSBuild
Select unique XElements (by attribute) with a filter using LinqToXml
Getting a unique id from a unix-like system
SQL Server - How to insert a record and make sure it is unique
C# equivalent of std::sort and std::unique
How to count unique records and get number of these uniques in table using SQL?
How To Create A 'Two-Sided' Unique Index On Two Fields?
What's the difference between creating a UNIQUE index as "index" or as "constraint" in SQL Server?
How to create a unique index on a NULL column?
MySQL: Select N rows, but with only unique values in one column
Extracting unique items from a list of mappings
Generating a unique ID in PHP
How do I get the unique elements from an array of hashes in Ruby?
Should I lock an ISAM table to insert a value into a unique key field?
Is the Hash of a GUID unique?
Uniq by object attribute in Ruby
Algorithm problem: letter combinations
Generating unique account numbers - recursive call
How can I get the unique values of an array in .net?
How do I remove duplicate items from an array in Perl?