views:

752

answers:

4

Hi, This question is more related to logic than any programming language. If the question is not apt for the forum please do let me know and I will delete this.

I have to write a logic to calculate scores for blogs for a Blog Award website. A blog may be nominated for multiple award categories and is peer-reviewed or rated by a Jury on a -1 to 5 scale (-1 to indicate a blog they utterly dislike). Now, a blog can be rated by one or more Jurors. One criterion while calculating final score for a blog is that if a blog is rated positively by more people it should get more weightage (and vice-versa). Similarly a blog rated -1 even by one Juror should have its score affected (-1 is sort of a Veto here). Lastly, I also want to have an additional score based on the Technorati rank of the blog (so that the final score is based on a mix of Juror rating + Technorati ranking).

Example: A blog is rated in category A by total 6 Jurors. 2 rate it at 3, 3 rate it at 2 and 1 rate it at 4. (I used to calculate the score as (2*3 + 3*2 + 1*4)/6 = 16/6 = 2.67 to get weighted average but I am not satisfied with this, primarily because it doesn't work well when a Juror rating is -1. Moreover, I need to add the Technorati ranking ranking criteria too) .

Could you help me decide the best way to calculate the final scores (keeping the rating method same as above as that cannot be changed now)?

A: 

you could internally work with scores from 0 to 6. Just do a shift by one, calculate the score and shift back. I guess the -1 has some disrupting effekt on your calculation.

Tobias Langner
+5  A: 

If you want to weight the effect of a -1 rating more strongly, use the same average score calculation but substitute -10 whenever you see -1. You can choose a value other than -10 if you don't want a negative rating to weight as strongly.

Greg Hewgill
Thanks Greg. What do you suggest about the Technorati ranking? And are you satisfied the way weighted average was being calculated?
Dee
You haven't specified what the Technorati ranking is. Is it also a value between -1 and 5? Or is it something else? How strongly do you want to weight it against the rankings of your jurors? These are the kinds of questions you will need to answer to determine how to use that ranking too.
Greg Hewgill
Greg, Technorati ranking is a score given by Technorati.com to a website based on various factors such as traffic, Linkback etc. Lower the rank value, better is the site. So a site with Technorati ranking 3000 is better off than one with a ranking 40000.
Dee
What is the range of Technorati values? It's also depends on how important you think the Technorati rating is. I mean you could just split how much each score matters, make the jurors account for some percentage of the score say 80% and then the Technorati rating could be the other 20%. But I don't really know how important you feel it should be. Also if you want to have more good scores count for more, just shift the score up based on the average score.
Jacob Schlather
+2  A: 

Calculating a score based on votes will be pretty easy. Adding the technorati rank will be the tricky part.

I made a quick script that calculates some scores based on this algorithm

score = ( vote_sum - ( vetos * veto_weight ) ) / number_of_votes

you can change the url paramters to get different values

There are a lot of ties, so maybe you could use technorati blog rank as a tie breaker

Galen
Thanks Galen for demonstrating it using a script.
Dee
+1 for pretty script :)
Mattis
A: 

You might look at using the lower bound of the Wilson score interval for your ratings.

See http://www.evanmiller.org/how-not-to-sort-by-average-rating.html for more details. Although, there, it is used for the simpler Bernoulli case.

The gist is if you have a lot of ratings you have a higher degree of confidence in your scoring. You can then combine the scores from your local ratings and the Technorati ratings, by weighting the scores by the number of voters locally and on Technorati.

As for wanting a single -1 vote to have high impact, just remap it to a large negative value proportional to your desired impact before feeding it into your scoring formula.

Edward Kmett