What should be considered when building a Recommendation Engine?

Question

I've read the book Programming Collective Intelligence and found it fascinating. I'd recently heard about a challenge amazon had posted to the world to come up with a better recommendation engine for their system.

The winner apparently produced the best algorithm by limiting the amount of information that was being fed to it.

As a first rule of thumb I guess... "More information is not necessarily better when it comes to fuzzy algorithms."

I know's it's subjective, but ultimately it's a measurable thing (clicks in response to recommendations).

Since most of us are dealing with the web these days and search can be considered a form of recommendation... I suspect I'm not the only one who'd appreciate other peoples ideas on this.

In a nutshell, "What is the best way to build a recommendation ?"

Answer 1

You don't want to use "overall popularity" unless you have no information about the user. Instead, you want to align this user with similar users and weight accordingly.

This is exactly what Bayesian Inference does. In English, it means adjusting the overall probability you'll like something (the average rating) with ratings from other people who generally vote your way as well.

Another piece of advice, but this time ad hoc: I find that there are people where if they like something I will almost assuredly not like it. I don't know if this effect is real or imagined, but it might be fun to build in a kind of "negative effect" instead of just clumping people by similarity.

Finally there's a company specializing in exactly this called SenseArray. The owner (Ian Clarke of freenet fame) is very approachable. You can use my name if you call him up.

Answer 2

There is an entire research area in computer science devoted to this subject. I'd suggest reading some articles.