Sort Best First

Question

Inspired by the Fastest Gun question on the central Meta, this script implements answer sorting based on a modified Wilson score. Installing the script adds a ‘Modified Wilson score’ sort option on question pages that attempts to sort the highest-quality answers first. One should hope this ordering method to boost hidden gems that would otherwise be buried under hastily-written popular-but-wrong answers that would otherwise hold the top spot under a naïve score tally sort.

Though initially directly inspired by the FGITW problem, ultimately the script aims to mitigate the related, but broader and more nebulous problem of popular-but-wrong (PBW) answers. As such, a number of weighing tweaks have been applied to take the specific voting structure of Stack Exchange sites into account:

• Downvotes are counted as five times heavier than upvotes, as they are assumed a better indicator of answer quality for a number of reasons:

  • The reputation threshold to cast upvotes is very low (15), much lower than for downvotes (125 on most sites). As such, upvotes should be assumed used more often by users who are less informed than the downvoters (with respect to the topic at hand, or the Stack Exchange customs).
  • Downvotes cost reputation to cast, therefore they are used more sparingly and with more care, and presumably, usually for reasons of merit.⁰ Upvotes are assumed more likely to be cast for more superficial reasons (appreciation of effort, formatting, writing style, humour).
  • Some users are reluctant to cast downvotes; a few have even explicitly stated that they never cast downvotes, no matter how bad an answer is (exhibit A; exhibit B; also, discussion). The weighing factor attempts to correct for this tendency, and estimate how everyone would have voted according to their convictions about the answer quality, regardless of any other considerations.

  To mitigate the effects of tactical and revenge downvoting, the first two downvotes are only weighed 1× and 2× as much as one upvote, respectively.

  The weighing is disabled completely on Stack Apps and Meta sites, including the central Meta; on those, the script reverts to using the plain, unweighed Wilson confidence interval. The above reasons don’t apply as much to Meta sites: on regular meta sites, neither upvotes nor downvotes cost reputation to cast, it is assumed most Meta participants can downvote, and furthermore it is assumed that Meta answers represent the users’ opinions, and so upvotes indicate agreement and cannot be meaningfully considered ‘incorrect’ (there is no such things as a popular-but-wrong meta post); the best the script can do is estimate how well a given answer would have been received, had everyone read it.

• Answers with more (unweighed) votes are judged more harshly: that is, the more votes an answer has relative to other answers, the closer the final score is to the lower end of its Wilson confidence interval. Answers with few votes have scores closer to the middle of the confidence interval; the answer with the most votes has the final score at the bottom of the confidence interval, unless all answers have fewer than 8 votes total (to avoid disturbing monotonicity where all answers have few votes overall).

• Bounties are counted with upvotes; to avoid high bounties biasing the scores too much, bounty awards are scaled and square-rooted. A bounty of 25 is worth 1 extra upvote; a bounty of 500 is worth 10 upvotes. each 25 points of bounty earned beyond the first is counted as one upvote. Bounties do not count towards harshness adjustment.

• Answer acceptance is not taken into account. In my experience, it tends to be a poor proxy for quality relative to other factors. Since askers usually upvote simultaneously with accepting the question anyway, I just did not bother accounting for acceptance at all.

If the question has answers that weren’t displayed on the current page, the script will generate stubs linking to the full answers. It may happen that those answers score higher than those visible on the current page and as such, the stubs may appear between full answers; this is intentional and not a bug. Deleted answers, when visible, are always sorted last.

The weighing factors were not determined by any principle other than ‘they felt right’, so beware. In particular, the 5× multiplier on downvotes was chosen mainly as a counterbalance to the five-times-smaller penalty for receiving an answer downvote than the reward for receiving an answer upvote. Statistical analyses of vote distribution, number of eligible voters and the number of views on each site might allow us to come up with fairer weighing factors.

---

⁰ Of course, downvoting for spite is not unheard of, though I assume most of it occurs as serial downvoting, which is usually swiftly corrected.

Script code

// ==UserScript==
// @name     StackExchange: Sort Best First
// @match    https://*.stackexchange.com/*
// @match    https://*.superuser.com/*
// @match    https://*.stackoverflow.com/*
// @match    https://*.mathoverflow.net/*
// @match    https://*.serverfault.com/*
// @match    https://*.askubuntu.com/*
// @match    https://stackapps.com/*
// @exclude  https://chat.stackexchange.com/*
// @exclude  https://api.stackexchange.com/*
// @exclude  https://data.stackexchange.com/*
// @exclude  https://openid.stackexchange.com/*
// @exclude  https://stackexchange.com/*
// @exclude  https://contests.stackoverflow.com/*
// @exclude  /^https?:\/\/winterbash\d{4,}\.stackexchange\.com\//
// @grant    GM.getValue
// @grant    GM.setValue
// @grant    GM.deleteValue
// ==/UserScript==

"use strict";

const OPT_UNCOVER_VOTES = false;

const GM_KEY_ACTIVE = `active-on:${location.origin}`;

const sleep = t =>
    new Promise((accept, reject) => setTimeout(accept, t));

const isMeta = /(^|\.)(meta\.|stackapps\.com$)/.test(location.hostname);

const GM_KEY_BACKOFF = 'backoff-until';

const getAnswerScores = async qid => {
  let backoffUntil = await GM.getValue(GM_KEY_BACKOFF, -Infinity);
  
  const usp = new URLSearchParams({
    site: location.hostname,
    filter: 'RonSjxB7EEr63nptQ05Wk',
  });

  const fullResult = [];

  let page = 1;
  for (let page = 1;; page++) {
    usp.set('page', page);
    backoffUntil = await GM.getValue(GM_KEY_BACKOFF, -Infinity);
    while (Date.now() < backoffUntil) {
      console.warn(`[SB1st] backing off until ${new Date(backoffUntil)} as told`);
      await sleep(backoffUntil - Date.now());
      backoffUntil = await GM.getValue(GM_KEY_BACKOFF, -Infinity);
    }
    GM.deleteValue(GM_KEY_BACKOFF, backoffUntil);
    const response = await fetch(`https://api.stackexchange.com/2.3/questions/${+qid}/answers?${usp}`);
    const result = await response.json();
    
    if (result.backoff) {
      console.warn(`[SB1st] Told to back off for ${result.backoff} s`);
      backoffUntil = Date.now() + result.backoff * 1000 + 500;
      GM.setValue(GM_KEY_BACKOFF, backoffUntil);
    }

    if (result.error_id)
      throw new Error(`${result.error_message} [${result.error_name}, ${result.error_id}]`);

    console.info(`[SB1st] Remaining API quota ${result.quota_remaining} of ${result.quota_max}`);

    fullResult.push(...result.items);

    if (!result.has_more)
      break;
  }

  return fullResult;
};

const scoreWilson = (answer) => {
  const up = answer.up_vote_count + Math.max((answer.awarded_bounty_amount ?? 0) / 25 - 1, 0);
  const dn = isMeta
        ? answer.down_vote_count
    : Math.max(0, answer.down_vote_count - 2) * 5
            + (answer.down_vote_count > 1) * 2
            + (answer.down_vote_count > 0);
  const total = up + dn;
  const harshness = answer.up_vote_count + answer.down_vote_count;
  const z = 1.96;

  let upper = 1, lower = 0;

  if (total !== 0) {
    const upRatio = up / total;
    const B = upRatio + z * z / (2 * total);
    const C = Math.sqrt((upRatio * (1 - upRatio) + z * z / (4 * total)) / total);
    const D = (1 + z * z / total);

    lower = (B - z * C) / D;
    upper = (B + z * C) / D;
  }

  return { lower, upper, total, harshness };
};

const qid = document.getElementById('question').dataset.questionid;
const answersList = document.getElementById('answers');

const keyFunc = key => (apple, orange) => {
  const appleKey = key(apple);
  const orangeKey = key(orange);
  return (appleKey > orangeKey) - (appleKey < orangeKey);
};

class RawHtmlString extends String {}

const html = (lits, ...values) =>
  new RawHtmlString(
    String.raw({ raw: lits.map(s => s.replace(/^\n[ \t]*|\n[ \t]*$/gu, '')) },
               ...values.map(value => value instanceof RawHtmlString
                                              ? value
                                                : String(value).replace(/[&<"']/gu, s => `&#${s.charCodeAt(0)};`))));

const performSort = async () => {
  const answerNodes = answersList.querySelectorAll('.answer');
  if (answerNodes.length < 2)
    return;

  const answerNodeMap = new Map(Array.from(answerNodes).map(node => [+node.dataset.answerid, node]));
  const unscoredAnswers = new Set(answerNodeMap.keys());
  const afterLastAnswer = answerNodes[answerNodes.length - 1].nextSibling;
  const answers = await getAnswerScores(qid);

  const scoredAnswers = answers.map(answer => {
    unscoredAnswers.delete(answer.answer_id);

    const { upper, lower, total, harshness } = scoreWilson(answer);
    let node = answerNodeMap.get(answer.answer_id);
    let anchorNode;

    if (node) {
      anchorNode = node.previousElementSibling;
      
      const voteCounter = node.querySelector('.js-vote-count');
      if (OPT_UNCOVER_VOTES) {
        const signChar = (n, sign) => 0 < n && n < 100 ? sign : "";
        voteCounter.innerHTML = html`
          <div class="fc-green-600">${signChar(answer.up_vote_count, "+")}${answer.up_vote_count}</div>
          <hr class="vote-count-separator">
          <div class="fc-red-600">${signChar(answer.down_vote_count, "−")}${answer.down_vote_count}</div>
        `;
      }
    } else {
      const by = (user) => user == null ? '' : html`
                    by ${
                user.user_id !== void 0
                ? html`
              <a href="/u/${user.user_id}">
                                <img src="${user.profile_image}" width="24" height="24" class="va-middle"
                     alt="">&nbsp;${user.display_name}
              </a>
                        `
                : html`${user.display_name}`
            }
                `;
      const at = (ts) => html`
                    <time title="${new Date(ts * 1000).toISOString()}">${
                new Intl.DateTimeFormat("en-US", {
              dateStyle: 'medium',
              timeStyle: 'short',
              hourCycle: 'h23',
            }).format(ts * 1000)
            }</time>
                `;
      
      anchorNode = document.createElement('a');
      anchorNode.name = answer.answer_id;

      node = document.createElement('div');
      node.dataset.answerid = answer.answer_id;
      node.dataset.ownerid = answer.owner.user_id;
      node.dataset.json = JSON.stringify(answer);
      node.className = 'answer';
      node.innerHTML = html`
        <div class="post-layout">
          <div class="votecell">
            <div class="grid fd-column ai-center ta-center">
              +${answer.up_vote_count}
              <hr class="vote-count-separator">
              −${answer.down_vote_count}
              ${
                answer.awarded_bounty_amount
                ? html`
                    <div>
                        <div class="s-badge s-badge__bounty s-badge__sm" style="margin-top: 0.75em;">+${answer.awarded_bounty_amount}</div>
                    </div>
                  `
                : ''
              }
            </div>
          </div>
          <div class="answercell">
            <div>
              <p style="margin-bottom: 0.3em; line-height: 28px;"><a href="/a/${answer.answer_id}">answer</a>
                            added ${at(answer.creation_date)} ${by(answer.owner)}
                            ${answer.community_owned_date !== void 0 ? " (community wiki)" : ""}
                            ${answer.last_edit_date
                            ? html`<p style="margin-bottom: 0.3em">last edited ${at(answer.last_edit_date)} ${by(answer.last_editor)}`
                            : ''}
                            ${answer.comment_count ? html`<p style="margin-bottom: 0.3em">${answer.comment_count} comment(s)` : ''}
            </div>
          </div>
        </div>
      `;
    }

    return {
      data: answer,
      node,
      anchorNode,
      wilsonInterval: [lower, upper],
      wilsonTotal: total,
      harshness,
    };
  });

  const maxHarshness = Math.max(
    8,
    ...scoredAnswers
    .map(answer => answer.harshness)
    .filter(total => total === total) /* filter out NaN */
  );

  scoredAnswers.forEach(answer => {
    const { harshness, wilsonInterval: [lower, upper], data: { creation_date } } = answer;

    const harshFactor = (harshness / maxHarshness) * 0.5 + 0.5;

    answer.harshFactor = harshFactor;
    answer.wilsonScore = (lower * harshFactor + upper * (1 - harshFactor));
  });

  scoredAnswers.sort(keyFunc(answer => -answer.wilsonScore));
  
  for (const id of unscoredAnswers)
    scoredAnswers.push({
      node: answerNodeMap.get(id),
      anchorNode: answerNodeMap.get(id).previousElementSibling,
    });

  /* maintain the scroll position over what the user is (presumably) currently reading */
  
  let targetNode = document.querySelector('a:target + .answer, :target');

  /* hack for comment links */
  if (!targetNode) {
    const m = /^#(?:comment(\d+)_(\d+))$/.exec(location.hash);
    /* picking .comment-text below because for some reason getBoundingClientRect() on the comment node itself returns [0, 0, 0, 0] */
    if (m) {
        targetNode = document.getElementById(`comment-${m[1]}`)?.querySelector('.comment-text');
      targetNode = targetNode ?? document.getElementById(`answer-${m[2]}`);
        }
  }

  if (targetNode) {
    const html = document.documentElement;
    const rect = targetNode.getBoundingClientRect();
    const { left, right, top, bottom } = rect;
    if (!rect || !(rect.bottom >= 0 && rect.right >= 0 && rect.left <= html.clientWidth && rect.top <= html.clientHeight))
      targetNode = null;
  }

  targetNode = targetNode ?? scoredAnswers.find(
    answer => {
      const { offsetTop: ny0, offsetHeight: nh } = answer.node;
      const { scrollY: sy, innerHeight: wh } = window;
      const whh = wh / 2;
      const ny1 = ny0 + nh;
      return (ny0 - sy <= whh) && (ny1 - sy >= whh);
    })?.node;

  const targetNodeDelta = targetNode ? window.scrollY - targetNode.offsetTop : null;
  
  for (const answer of scoredAnswers) {
    const { node, anchorNode } = answer;

    afterLastAnswer.parentNode.insertBefore(node, afterLastAnswer);
    afterLastAnswer.parentNode.insertBefore(anchorNode, node);
  }
  
  if (targetNode) {
    window.scroll({ top: targetNode.offsetTop + targetNodeDelta });
  }
};

const sortSwitch = document.querySelector('select#answer-sort-dropdown-select-menu');
const sortReset = document.querySelector('label[for="answer-sort-dropdown-select-menu"] + .js-sort-preference-change');
let option = null;

const switchSort = async () => {
  const waitNode = answersList;

  waitNode.style.cursor = 'wait';
  waitNode.style.opacity = '0.5';
  try {
    await performSort();
  } catch (e) {
    alert(`Sort Best First failed: ${e}`);
    throw e;
  } finally {
    waitNode.style.cursor = '';
    waitNode.style.opacity = '';
  }

  if (option) {
    option.selected = true;
  }
  sortReset?.classList.remove('d-none');

  // hack to make links keep working
  try {
    const n = new URL(location.href);
    if (!n.searchParams.get('answertab'))
      return;
    n.searchParams.delete('answertab');
    const s = n?.toString().replace(/#.*$/u, "");
    history.replaceState(history.state, '', s);
  } catch (e) {
    console.warn(`[SB1st] error while resetting ?answertab=`, e);
  }
};

if (sortSwitch) {
  option = document.createElement('option');
  option.textContent = isMeta
    ? 'Wilson score (best first)'
    : 'Modified Wilson score (best first)';
  sortSwitch.addEventListener('change', async ev => {
    if (option.selected) {
      ev.stopPropagation();
      await switchSort();
    }
    GM.setValue(GM_KEY_ACTIVE, option.selected);
  }, false);
  sortSwitch.appendChild(option);
}

sortReset?.addEventListener('click', async ev => {
  if (ev.defaultPrevented)
    return;
  GM.setValue(GM_KEY_ACTIVE, false);
}, false);

if (answersList) {
    if (!new URL(location.href).searchParams.has('answertab')) {
    GM.getValue(GM_KEY_ACTIVE, false).then(value => {
      if (value)
        switchSort();
    });
  }
}

Answer 1

Test cases:

• The good: Much of the time, vote-weighing order successfully puts first a better-quality answer than the most popular one as measured by naïve score sort.

  • Under “Passing a string literal as a type argument to a class template”, the “Votes” order currently places an answer with a not-quite-solution workaround (+56 / −2) first. The “Best” order by this script places an answer with a real solution, with some pointers to language standardization efforts (+22 / −0) first. The latter I would have found pretty hard to discover otherwise; I count this one as a success.
  • Under “Why is it string.join(list) instead of list.join(string)?”, the script puts an answer that points to the actual mailing list discussions where this was decided (+381 / −0) at the top, above the top-voted just-so story that doesn’t actually justify the choice (+1376 / −5). I think this one counts as a success as well.
  • This Stack Overflow Meta answer about canonical duplicates points to two questions, where the naïve incumbents are substandard or outright misleading. My script does much better at surfacing higher-quality answers:
    • For “Which functions from the standard library must (should) be avoided?”, the naïve incumbent by @Michael Aaron Safyan (+65 / −5) recommends replacing the str* family of functions with their strn* counterparts, which are actually rather problematic in their own right. Modified Wilson score places the answer by @Dipstick (+24 / −0) which points this out first, and another such by @Michael Burr (+20 / −0) second, while the naïve incumbent is placed seventh; the meta answerer’s own pretty comprehensive answer (+13 / −2) is placed ninth. One can argue this is a borderline “meh” case (see below), but I still consider it an improvement over naïve scoring.
    • Under “Why are these constructs using pre and post-increment undefined behavior?”, the naïve incumbent is a vague answer barely saying “it’s UB” with some Wikipedia links (+622 / −15, +500 bounty). Weighed Wilson score places @haccks’s answer that points to specific passages in the standard (+86 / −0) first, and the naïve incumbent seventh.
  • On the central Meta question “What's the official SE response to serious mentions of suicide or self-harm in posts?”, the naïve incumbent full of canned responses (+109 / −9, +100 bounty) appears above an answer explaining why those are positively harmful (+68 / −2, +150 bounty). Wilson score puts them in the opposite order, despite the script not applying the usual vote weighing.
  • Under “Why is appending to the innerHTML property bad?”, the accepted naïve incumbent (+73 / −6) is a relatively superficial answer, focusing only on the performance aspect. “Best” sort instead places a much more comprehensive answer by Sheepy (+44 / −0) first – and did so even before I decided to award it a bounty.

  At least for me, as of April 2022, the much-touted “Trending” order often still puts the naïve incumbents first, though the “Best” answer sometimes comes second.

• The bad: In a few cases, I have noticed weighing to be actively harmful.

  • Under “Is RefCell an appropriate workaround to borrow two mutable elements from a vector?”, my answer (+3 / −0) appears below the accepted answer (+1 / −0). The accepted answer is perfectly fine, but the fact that apparently the harshness adjustment (picking lower points from the confidence interval for answers with more votes) makes scores non-monotonic is a little concerning. Maybe I should tweak the adjustment or ditch it altogether.

    I added a simple artificial correction that ensures harshness adjustment keeps sorting monotone if all answers have few votes.

  • Weighted sorting on “How do I return the response from an asynchronous call?” is pretty bad, to the point where naïve sort is superior: it overweighs downvotes on the top-scoring answer by Felix King (+6516 / −16, +150 bounty) so much that it appears third on the list, even though there is nothing particularly wrong with it (other than being somewhat long-winded); weighted sort instead prefers the answer by T.J. Crowder (+182 / −0), which is unfortunately focused on a specific sub-instance of the problem, and probably wasn’t intended as the “main” answer even by its author. Under current weighting, this ordering can only be corrected by tactical downvoting.

  • Under “How do I test for an empty JavaScript object?”, naïve score sort prefers the accepted answer (+7384 / −12) listing pure-JS and a number of library solutions. Weighted Wilson score instead chooses a benchmark answer (+362 / −1). The benchmarks in the latter were pretty sloppy and in the end rather meaningless, which I was apparently the first to notice. Meanwhile, there is nothing really wrong with the accepted top-scoring answer. It seems negative votes on the top-scoring answer are overweighed just like in the previous case study.

  • Under “What is the point of void operator in JavaScript?”, both naïve score sort and weighted Wilson sort prefer @Crescent Fresh’s answer (+65 / −1), which is not actually bad on its own, but is not very in-depth either. However, in the second place there is a completely bogus answer by @Ben Aston (+38 / −2) that refers to a non-existent version of the ECMAScript specification. In the third place, there is a pretty decent accepted answer by @Joseph Kwan (+24 / −2), which weighted Wilson sort actually moves one place down in the list.

• The meh: Vote weighing can improve the visibility of better-quality answers, but it’s not a panacea, and the results can be underwhelming – it doesn’t place the best answer first every single time. This often happens when insufficient votes are cast against the popular-but-flawed answer, and it cannot be corrected without applying considerably more drastic, and therefore dubious, weighing factors. Examples of this:

  • Under “What algorithm did Microsoft use to dither colour in early versions of Windows?”, which I dare claim to be probably one of the worst victims of the fastest-gun problem, my own in-depth researched answer (+24 / −0) appears second, while the accepted misleading answer (+171 / −1) appears first, just like with naïve scoring. I don’t think any other scoring method could do much better here, though. There is only so much information that can be drawn from vote counts alone. For my answer to rise to the top under the “Best” order would require a large number of people to vote down answers simply for being incomplete, or for being mostly correct but subtly misleading; I doubt that many are particularly motivated to do so. As of November 2022, my answer (+33 / −0) is now sorted above the naïve incumbent (+171 / −2) under the “Best” order.
  • Under “Reverse a string in Python”, most answers are based on reversing individual code points, including the naïve incumbent (+3007 / −5). The only answer pointing out that this is incorrect (+43 / −0) and offering an alternative based on grapheme clusters appears fourth in the naïve score sort and second under the “Best” sort, behind the naïve incumbent. Again, the incumbent is probably not going to be displaced without more votes being cast against it, and in favour of the other answer. (This remains the case after I awarded a bounty to the latter answer.)
  • Under “What does "i" represent in Python .pyi extension?”, both naïve and weighted sort place the a speculative, meandering accepted answer (+139 / −2) first, and an evidence-based answer by @Pyprohly (+29 / −1) third.