Test cases:
The good: Much of the time, vote-weighing order successfully puts first a better-quality answer than the most popular one as measured by naïve score sort.
- Under ‘Passing a string literal as a type argument to a class template’, the ‘Votes’ order currently places an answer with a not-quite-solution workaround (+56 / −2) first. The ‘Best’ order by this script places an answer with a real solution, with some pointers to language standardization efforts (+22 / −0) first. The latter I would have found pretty hard to discover otherwise; I count this one as a success.
- Under ‘Why is it string.join(list) instead of list.join(string)?’, the script puts an answer that points to the actual mailing list discussions where this was decided (+381 / −0) at the top, above the top-voted just-so story that doesn’t actually justify the choice (+1376 / −5). I think this one counts as a success as well.
- This Stack Overflow Meta answer about canonical duplicates points to two questions, where the naïve incumbents are substandard or outright misleading. My script does much better at surfacing higher-quality answers:
- For ‘Which functions from the standard library must (should) be avoided?’, the naïve incumbent by @Michael Aaron Safyan (+65 / −5) recommends replacing the
str*family of functions with theirstrn*counterparts, which are actually rather problematic in their own right. Modified Wilson score places the answer by @Dipstick (+24 / −0) which points this out first, and another such by @Michael Burr (+20 / −0) second, while the naïve incumbent is placed seventh; the meta answerer’s own pretty comprehensive answer (+13 / −2) is placed ninth. One can argue this is a borderline ‘meh’ case (see below), but I still consider it an improvement over naïve scoring. - Under ‘Why are these constructs using pre and post-increment undefined behavior?’, the naïve incumbent is a vague answer barely saying ‘it’s UB’ with some Wikipedia links (+622 / −15, +500 bounty). Weighed Wilson score places @haccks’s answer that points to specific passages in the standard (+86 / −0) first, and the naïve incumbent seventh.
- For ‘Which functions from the standard library must (should) be avoided?’, the naïve incumbent by @Michael Aaron Safyan (+65 / −5) recommends replacing the
- On the central Meta question ‘What's the official SE response to serious mentions of suicide or self-harm in posts?’, the naïve incumbent full of canned responses (+109 / −9, +100 bounty) appears above an answer explaining why those are positively harmful (+68 / −2, +150 bounty). Wilson score puts them in the opposite order, despite the script not applying the usual vote weighing.
At least for me, as of April 2022, the much-touted ‘Trending’ order still puts the naïve incumbents first, though the ‘Best’ answer often comes second.
The bad: In a few cases, I have noticed weighing to be actively harmful.
Under ‘Is RefCell an appropriate workaround to borrow two mutable elements from a vector?’, my answer (+3 / −0) appears below the accepted answer (+1 / −0). The accepted answer is perfectly fine, but the fact that apparently the harshness adjustment (picking lower points from the confidence interval for answers with more votes) makes scores non-monotonic is a little concerning. Maybe I should tweak the adjustment or ditch it altogether.I added a simple artificial correction that ensures harshness adjustment keeps sorting monotone if all answers have few votes.
The meh: Vote weighing can improve the visibility of better-quality answers, but it’s not a panacea, and it doesn’t always place the best answer first. This often happens when insufficient votes are cast against the popular-but-flawed answer, and it cannot be corrected without applying considerably more drastic, and therefore dubious, weighing factors. Examples of this:
Under ‘What algorithm did Microsoft use to dither colour in early versions of Windows?’, which I dare claim to be probably one of the worst victims of the fastest-gun problem, my own in-depth researched answer (+24 / −0) appears second, while the accepted misleading answer (+171 / −1) appears first, just like with naïve scoring. I don’t think any other scoring method could do much better here, though. There is only so much information that can be drawn from vote counts alone. For my answer to rise to the top under the ‘Best’ order would require a large number of people to vote down answers simply for being incomplete, or for being mostly correct but subtly misleading; I doubt that many are particularly motivated to do so.As of November 2022, my answer (+33 / −0) is now sorted above the naïve incumbent (+171 / −2) under the ‘Best’ order.- Under ‘Reverse a string in Python’, most answers are based on reversing individual code points, including the naïve incumbent (+3007 / −5). The only answer pointing out that this is incorrect (+43 / −0) and offering an alternative based on grapheme clusters appears fourth in the naïve score sort and second under the ‘Best’ sort, behind the naïve incumbent. Again, the incumbent is probably not going to be displaced without more votes being cast against it, and in favour of the other answer. (This remains the case after I awarded a bounty to the latter answer.)
- Under ‘What does "i" represent in Python .pyi extension?’, both naïve and weighted sort place the a speculative, meandering accepted answer (+139 / −2) first, and an evidence-based answer by @Pyprohly (+29 / −1) third.