I came across the following blog post on the Internet recently: How Not to Sort by Average Rating:
PROBLEM: You are a web programmer. You have users. Your users rate stuff on your site. You want to put the highest-rated stuff at the top and lowest-rated at the bottom. You need some sort of “score” to sort by.
WRONG SOLUTION #1: Score = (Positive ratings) − (Negative ratings)
Why it is wrong: Suppose one item has 600 positive ratings and 400 negative ratings: 60% positive. Suppose item two has 5,500 positive ratings and 4,500 negative ratings: 55% positive. This algorithm puts item two (score = 1000, but only 55% positive) above item one (score = 200, and 60% positive). WRONG.
Sites that make this mistake: Urban Dictionary[, Stack Exchange]
Of course, I was intrigued since Stack Exchange uses that exact rating system. The article then describes another wrong solution (average ratings), which is used by Amazon among others.
At the end of the post, it states the following "correct" solution, which not only takes the rating proportions into account, but also the overall number of ratings, to produce a statistical probability of how well-received a given post is:
CORRECT SOLUTION: Score = Lower bound of Wilson score confidence interval for a Bernoulli parameter
Say what: We need to balance the proportion of positive ratings with the uncertainty of a small number of observations. Fortunately, the math for this was worked out in 1927 by Edwin B. Wilson. What we want to ask is: Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what? Wilson gives the answer. Considering only positive and negative ratings (i.e. not a 5-star scale), the lower bound on the proportion of positive ratings is given by: [formula]
(I won't go into the full statistical marks as to how exactly it works; if you're into that stuff, you can click the link above to read the full math behind it.)
Following along with the formula and code examples given there, I've written the following JS snippet that calculates a post's Wilson confidence rating given its numbers of upvotes and downvotes:
function calculateWilsonRating(upvotes, downvotes) {
var total = upvotes + downvotes;
if(total == 0)
return 0;
var z = 1.96; //confidence Z-score for 95% confidence; for 99% confidence use 2.58
var p = upvotes / total;
return (p + z * z / (2 * total)
- z * Math.sqrt((p * (1 - p) + (z * z) / (4 * total)) / total))
/ (1 + (z * z) / total);
}
I'd like a user script that provides a handy button to calculate this for a given post (question or answer). Once the button is clicked, the script should then use the Stack Exchange API to fetch the up and down votes for a given post (i.e. it should be compatible for all users regardless of reputation), plugs in the result into that function, and displays the result on that post.
(Due to API quota restrictions, I only want the algorithm to be run upon the clicking of a button.)
