Best way to detect and store path combinations for analysing purpose later

Question

I am searching for ideas/examples on how to store path patterns from users - with the goal of analysing their behaviours and optimizing on "most used path" when we can detect them somehow.

Eg. which action do they do after what, so that we later on can check to see if certain actions are done over and over again - therefore developing a shortcut or assembling some of the actions into a combined multiaction.

My first guess would be some sort of "simple log", perhaps stored in some SQL-manner, where we can keep each action as an index and then just record everything.

Problem is that the path/action might be dynamically changed - even while logging - so we need to be able to take care of this fact too, when looking for patterns later.

Would you log everthing "bigtime" first and then POST-process every bit of details after some time or do you have great experience with other tactics?

My worry is that this is going to take up space, BIG TIME while logging 1000 users each day for a month or more.

Hope this makes sense and I am curious to see if anyone can provide sample code, pseudocode or perhaps links to something usefull.

Our tools will be C#, SQL-database, XML and .NET 3.5 - clients could also get .NET 4.0 if needed.

Patterns examples as we expect them

...
User #1001: A-B-A-A-A-B-C-E-F-G-H-A-A-A-C-B-A
User #1002: B-A-A-B-C-E-F
User #1003: F-B-B-A-E-C-A-A-A   
User #1002: C-E-F
...

etc. no real way to know what they do next nor how many they will use, how often they will do it.

A secondary goal, if possible, if we later on add a new "action" called G (just sample to illustrate, there will be hundreds of actions) how could we detect these new behaviours influence on the previous patterns.

To explain it better, my thought here would be some way to detect "patterns within patterns", sort of like how compressions work, so that "repeative patterns" are spottet. We dont know how long these patterns might be, nor how often they might come. How do we break this down into "small bits and pieces" - whats the best approach you think?

Answer 1

I am not sure what you mean by path, but, if you gave every action in a path a unique symbol, you could reduce the problem to longest common substring or subsequence.

Or have a map of paths to the number of times that action occurred. Every time a certain path happens, increment the count for that path. Then sort to find the most common.