Sort Best First

Question

Inspired by the Fastest Gun question on the central Meta, this script implements answer sorting based on a modified Wilson score. Installing the script adds a ‘Modified Wilson score’ sort option on question pages that attempts to sort the highest-quality answers first. One should hope this ordering method to boost hidden gems that would otherwise be buried under hastily-written popular-but-wrong answers that would otherwise hold the top spot under a naïve score tally sort.

Though initially directly inspired by the FGITW problem, ultimately the script aims to mitigate the related, but broader and more nebulous problem of popular-but-wrong (PBW) answers. As such, a number of weighing tweaks have been applied to take the specific voting structure of Stack Exchange sites into account:

• Downvotes are counted as five times heavier than upvotes, as they are assumed a better indicator of answer quality for a number of reasons:

  • The reputation threshold to cast upvotes is very low (15), much lower than for downvotes (125 on most sites). As such, upvotes should be assumed used more often by users who are less informed than the downvoters (with respect to the topic at hand, or the Stack Exchange customs).
  • Downvotes cost reputation to cast, therefore they are used more sparingly and with more care, and presumably, usually for reasons of merit.⁰ Upvotes are assumed more likely to be cast for more superficial reasons (appreciation of effort, formatting, writing style, humour).

  This weighing is disabled on Stack Apps and Meta sites, including the central Meta; on those, the script reverts to using the plain, unweighed Wilson confidence interval. The above reasons don’t apply as much to Meta sites: on regular meta sites, neither upvotes nor downvotes cost reputation to cast, it is assumed most Meta participants can downvote and furthermore it is assumed that Meta answers represent the users’ opinions, where votes cannot be meaningfully ‘incorrect’ in judging an answer’s merit (there is no such things as a popular-but-wrong meta post); the best the script can do is estimate how well a given answer would have been received, had everyone read it.

• Bounties are counted with upvotes; to avoid high bounties biasing the scores too much, bounty awards are scaled and square-rooted. A bounty of 25 is worth 1 extra upvote; a bounty of 500 is worth 10 upvotes.

• Answers with more (unweighed) votes are judged more harshly: that is, the more votes an answer has relative to other answers, the closer the final score is to the lower end of its Wilson confidence interval. Answers with few votes have scores closer to the middle of the confidence interval; the answer with the most votes has the final score at the bottom of the confidence interval, unless all answers have fewer than 8 votes total (to avoid disturbing monotonicity where all answers have few votes overall).

• Answer acceptance is not taken into account. In my experience, it tends to be a poor proxy for quality relative to other factors. Since askers usually upvote simultaneously with accepting the question anyway, I just did not bother accounting for acceptance at all.

If the question has answers that weren’t displayed on the current page, the script will generate stubs linking to the full answers. It may happen that those answers score higher than those visible on the current page and as such, the stubs may appear between full answers; this is intentional and not a bug. Deleted answers, when visible, are always sorted last.

The weighing factors were not determined by any principle other than ‘they felt right’, so beware. In particular, the 5× multiplier on downvotes was chosen mainly as a counterbalance to the five-times-smaller penalty for receiving an answer downvote than the reward for receiving an answer upvote. Statistical analyses of vote distribution, number of eligible voters and the number of views on each site might allow us to come up with fairer weighing factors.

---

⁰ Of course, downvoting for spite is not unheard of, though I assume most of it occurs as serial downvoting, which is usually swiftly corrected.

Script code

// ==UserScript==
// @name     StackExchange: Sort Best First
// @grant    none
// @match    https://*.stackexchange.com/*
// @match    https://*.superuser.com/*
// @match    https://*.stackoverflow.com/*
// @match    https://*.mathoverflow.net/*
// @match    https://*.serverfault.com/*
// @match    https://*.askubuntu.com/*
// @match    https://stackapps.com/*
// @exclude  https://chat.stackexchange.com/*
// @exclude  https://api.stackexchange.com/*
// @exclude  https://data.stackexchange.com/*
// @exclude  https://openid.stackexchange.com/*
// @exclude  https://stackexchange.com/*
// @exclude  https://contests.stackoverflow.com/*
// @exclude  /^https?:\/\/winterbash\d{4,}\.stackexchange\.com\//
// ==/UserScript==

const sleep = t =>
    new Promise((accept, reject) => setTimeout(accept, t));

const isMeta = /(^|\.)(meta\.|stackapps\.com$)/.test(location.hostname);

const getAnswerScores = async qid => {
  const usp = new URLSearchParams;
  usp.set('site', location.hostname);
  usp.set('filter', '!2oRRX0IVTN-3*1DAJ2N2SJiLCR.2VfC)CoC(Q7r-p.');

  const fullResult = [];

  let page = 1;
  for (let page = 1;; page++) {
    usp.set('page', page);
    const response = await fetch(`https://api.stackexchange.com/2.2/questions/${qid}/answers?${usp}`);
    const result = await response.json();

    if (result.error_id)
      throw new Error(`${result.error_message} [${result.error_name}, ${result.error_id}]`);

    fullResult.push(...result.items);

    if (!result.has_more)
      break;
    if (result.backoff)
      await sleep(result.backoff * 1000);
  }

  return fullResult;
};

const scoreWilson = (answer) => {
  const SCALE = Math.log(10) / Math.log(20);
  const up = answer.up_vote_count + (answer.awarded_bounty_amount ? Math.pow(answer.awarded_bounty_amount / 25, SCALE) : 0);
  const dn = isMeta ? answer.down_vote_count : 5 * answer.down_vote_count;
  const total = up + dn;
  const harshness = answer.up_vote_count + answer.down_vote_count;
  const z = 1.96;

  let upper = 1, lower = 0;

  if (total !== 0) {
    const upRatio = up / total;
    const B = upRatio + z * z / (2 * total);
    const C = Math.sqrt((upRatio * (1 - upRatio) + z * z / (4 * total)) / total);
    const D = (1 + z * z / total);

    lower = (B - z * C) / D;
    upper = (B + z * C) / D;
  }

  return { lower, upper, total, harshness };
};

const qid = document.getElementById('question').dataset.questionid;
const answersList = document.getElementById('answers');

const keyFunc = key => (apple, orange) => {
  const appleKey = key(apple);
  const orangeKey = key(orange);
  return (appleKey > orangeKey) - (appleKey < orangeKey);
};

const rawHTML = s => {
  const result = new String(s);
  result.rawHTML = true;
  return result;
};

const html = (lits, ...values) => {
  const ESCAPE = { '<': '&lt;', '&': '&amp;', '"': '&quot;', "'": '&#39;', };
  return rawHTML(String.raw({ raw: lits }, ...values.map(value => {
    if (value instanceof Node)
      return (new XMLSerializer).serializeToString(value);
    else if (value instanceof String && value.rawHTML)
      return value;
    else
      return String(value).replace(/[&<"']/gu, s => ESCAPE[s]);
  })));
};

const performSort = async () => {
  const answerNodes = answersList.querySelectorAll('.answer');
  if (answerNodes.length < 2)
    return;

  const answerNodeMap = new Map(Array.from(answerNodes).map(node => [+node.dataset.answerid, node]));
  const unscoredAnswers = new Set(answerNodeMap.keys());
  const afterLastAnswer = answerNodes[answerNodes.length - 1].nextSibling;
  const answers = await getAnswerScores(qid);

  const scoredAnswers = answers.map(answer => {
    unscoredAnswers.delete(answer.answer_id);

    const { upper, lower, total, harshness } = scoreWilson(answer);
    let node = answerNodeMap.get(answer.answer_id);
    let anchorNode;

    if (node)
      anchorNode = node.previousElementSibling;
    else {
      anchorNode = document.createElement('a');
      anchorNode.name = answer.answer_id;

      node = document.createElement('div');
      node.dataset.answerid = answer.answer_id;
      node.dataset.ownerid = answer.owner.user_id;
      node.className = 'answer';
      node.innerHTML = html`
        <div class="post-layout">
          <div class="votecell">
            <div class="grid fd-column ai-center ta-center">
              +${answer.up_vote_count}
              <hr class="vote-count-separator">
              −${answer.down_vote_count}
            </div>
          </div>
          <div class="answercell">
            <div>
              <a href="/a/${answer.answer_id}">answer</a>
              by ${
                answer.owner.user_id !== void 0
                  ? html`
                    <a href="/u/${answer.owner.user_id}">
                      <img src="${answer.owner.profile_image}" width="32" height="32" alt="" class="va-middle">
                      ${answer.owner.display_name}
                    </a>
                  `
                  : html`${answer.owner.display_name}`
              }
            </div>
          </div>
        </div>
      `;
    }

    return {
      data: answer,
      node,
      anchorNode,
      wilsonInterval: [lower, upper],
      wilsonTotal: total,
      harshness,
    };
  });

  const maxHarshness = Math.max(
    8,
    ...scoredAnswers
    .map(answer => answer.harshness)
    .filter(total => total === total) /* filter out NaN */
  );

  scoredAnswers.forEach(answer => {
    const { harshness, wilsonInterval: [lower, upper], data: { creation_date } } = answer;

    const harshFactor = (harshness / maxHarshness) * 0.5 + 0.5;

    answer.harshFactor = harshFactor;
    answer.wilsonScore = (lower * harshFactor + upper * (1 - harshFactor));
    
    answer.node.dataset.sbfData = JSON.stringify({
      lower, upper, harshness
    });
  });

  scoredAnswers.sort(keyFunc(answer => -answer.wilsonScore));
  
  for (const id of unscoredAnswers)
    scoredAnswers.push({
      node: answerNodeMap.get(id),
      anchorNode: answerNodeMap.get(id).previousElementSibling,
    });

  /* maintain the scroll position over what the user is (presumably) currently reading */
  
  let targetNode = document.querySelector('a:target + .answer, :target');

  /* hack for comment links */
  if (!targetNode) {
    const m = /^#(comment(\d+)_(\d+))$/.exec(location.hash);
    /* picking .comment-text below because for some reason getBoundingClientRect() on the comment node itself returns [0, 0, 0, 0] */
    if (m) {
        targetNode = document.getElementById(`comment-${m[2]}`)?.querySelector('.comment-text');
      targetNode = targetNode ?? document.getElementById(`answer-${m[3]}`);
        }
  }

  if (targetNode) {
    const html = document.documentElement;
    const rect = targetNode.getBoundingClientRect();
    const { left, right, top, bottom } = rect;
    if (!rect || !(rect.bottom >= 0 && rect.right >= 0 && rect.left <= html.clientWidth && rect.top <= html.clientHeight))
      targetNode = null;
  }

  targetNode = targetNode ?? scoredAnswers.find(
    answer => {
      const { offsetTop: ny0, offsetHeight: nh } = answer.node;
      const { scrollY: sy, innerHeight: wh } = window;
      const whh = wh / 2;
      const ny1 = ny0 + nh;
      return (ny0 - sy <= whh) && (ny1 - sy >= whh);
    })?.node;

  const targetNodeDelta = targetNode ? window.scrollY - targetNode.offsetTop : null;
  
  for (const answer of scoredAnswers) {
    const { node, anchorNode } = answer;

    afterLastAnswer.parentNode.insertBefore(node, afterLastAnswer);
    afterLastAnswer.parentNode.insertBefore(anchorNode, node);
  }
  
  if (targetNode) {
    window.scroll({ top: targetNode.offsetTop + targetNodeDelta });
  }
};

const switchSort = async () => {
  const waitNode = answersList;
  waitNode.style.cursor = 'wait';
  waitNode.style.opacity = '0.5';

  try {
    await performSort();
  } catch (e) {
    alert(`Sort Best First failed: ${e}`);
    throw e;
  } finally {
    waitNode.style.cursor = '';
    waitNode.style.opacity = '';
  }

  if (option) {
    option.selected = true;
  }

  // hack to make links keep working
  try {
    const n = new URL(window.location.href);
    if (!n.searchParams.get('answertab'))
      return;
    n.searchParams.delete("answertab");
    const s = n?.toString().replace(/#.*$/u, "");
    history.replaceState(history.state, '', s);
  } catch (e) {
    console.warn(e);
  }
};

const LSKEY = 'sbf-active';

const sortSwitch = document.querySelector('select#answer-sort-dropdown-select-menu');
let option = null;

if (sortSwitch) {
  option = document.createElement('option');
  option.textContent = isMeta
    ? 'Wilson score (best first)'
    : 'Modified Wilson score (best first)';
  sortSwitch.addEventListener('change', async ev => {
    if (option.selected) {
      ev.stopPropagation();
      await switchSort();
    }
    localStorage.setItem(LSKEY, option.selected ? '1' : '0');
  }, true);
  sortSwitch.appendChild(option);
}

if (answersList) {
  let doIt = false;

  try {
    doIt = (localStorage.getItem(LSKEY) === '1');
  } catch (e) {
    console.warn(e);
  }

  if (doIt)
    switchSort();
}

Answer 1

Test cases:

• The good: Much of the time, vote-weighing order successfully puts first a better-quality answer than the most popular one as measured by naïve score sort.

  • Under ‘Passing a string literal as a type argument to a class template’, the ‘Votes’ order currently places an answer with a not-quite-solution workaround (+56 / −2) first. The ‘Best’ order by this script places an answer with a real solution, with some pointers to language standardization efforts (+22 / −0) first. The latter I would have found pretty hard to discover otherwise; I count this one as a success.
  • Under ‘Why is it string.join(list) instead of list.join(string)?’, the script puts an answer that points to the actual mailing list discussions where this was decided (+381 / −0) at the top, above the top-voted just-so story that doesn’t actually justify the choice (+1376 / −5). I think this one counts as a success as well.
  • This Stack Overflow Meta answer about canonical duplicates points to two questions, where the naïve incumbents are substandard or outright misleading. My script does much better at surfacing higher-quality answers:
    • For ‘Which functions from the standard library must (should) be avoided?’, the naïve incumbent by @Michael Aaron Safyan (+65 / −5) recommends replacing the str* family of functions with their strn* counterparts, which are actually rather problematic in their own right. Modified Wilson score places the answer by @Dipstick (+24 / −0) which points this out first, and another such by @Michael Burr (+20 / −0) second, while the naïve incumbent is placed seventh; the meta answerer’s own pretty comprehensive answer (+13 / −2) is placed ninth. One can argue this is a borderline ‘meh’ case (see below), but I still consider it an improvement over naïve scoring.
    • Under ‘Why are these constructs using pre and post-increment undefined behavior?’, the naïve incumbent is a vague answer barely saying ‘it’s UB’ with some Wikipedia links (+622 / −15, +500 bounty). Weighed Wilson score places @haccks’s answer that points to specific passages in the standard (+86 / −0) first, and the naïve incumbent seventh.
  • On the central Meta question ‘What's the official SE response to serious mentions of suicide or self-harm in posts?’, the naïve incumbent full of canned responses (+109 / −9, +100 bounty) appears above an answer explaining why those are positively harmful (+68 / −2, +150 bounty). Wilson score puts them in the opposite order, despite the script not applying the usual vote weighing.

  At least for me, as of April 2022, the much-touted ‘Trending’ order still puts the naïve incumbents first, though the ‘Best’ answer often comes second.

• The bad: In a few cases, I have noticed weighing to be actively harmful.

  • Under ‘Is RefCell an appropriate workaround to borrow two mutable elements from a vector?’, my answer (+3 / −0) appears below the accepted answer (+1 / −0). The accepted answer is perfectly fine, but the fact that apparently the harshness adjustment (picking lower points from the confidence interval for answers with more votes) makes scores non-monotonic is a little concerning. Maybe I should tweak the adjustment or ditch it altogether.

    I added a simple artificial correction that ensures harshness adjustment keeps sorting monotone if all answers have few votes.

  • Weighted sorting on ‘How do I return the response from an asynchronous call?’ is pretty bad, to the point where naïve sort is superior: it overweighs downvotes on the top-scoring answer by Felix King (+6516 / −16, +150 bounty) so much that it appears third on the list, even though there is nothing particularly wrong with it (other than being somewhat long-winded); weighted sort instead prefers the answer by T.J. Crowder (+182 / −0), which is unfortunately focused on a specific sub-instance of the problem, and probably wasn’t intended as the ‘main’ answer even by its author. Under current weighing, this ordering can only be corrected by tactical downvoting.

  • Under ‘How do I test for an empty JavaScript object?’, naïve score sort prefers the accepted answer (+7384 / −12) listing pure-JS and a number of library solutions. Weighted Wilson score instead chooses a benchmark answer (+362 / −1). The benchmarks in the latter were pretty sloppily and in the end rather meaningless, which I was apparently the first to notice. Meanwhile, there is nothing really wrong with the accepted top-scoring answer. It seems negative votes on the top-scoring answer are overweighed just like in the previous case study.

• The meh: Vote weighing can improve the visibility of better-quality answers, but it’s not a panacea, and it doesn’t always place the best answer first. This often happens when insufficient votes are cast against the popular-but-flawed answer, and it cannot be corrected without applying considerably more drastic, and therefore dubious, weighing factors. Examples of this:

  • Under ‘What algorithm did Microsoft use to dither colour in early versions of Windows?’, which I dare claim to be probably one of the worst victims of the fastest-gun problem, my own in-depth researched answer (+24 / −0) appears second, while the accepted misleading answer (+171 / −1) appears first, just like with naïve scoring. I don’t think any other scoring method could do much better here, though. There is only so much information that can be drawn from vote counts alone. For my answer to rise to the top under the ‘Best’ order would require a large number of people to vote down answers simply for being incomplete, or for being mostly correct but subtly misleading; I doubt that many are particularly motivated to do so. As of November 2022, my answer (+33 / −0) is now sorted above the naïve incumbent (+171 / −2) under the ‘Best’ order.
  • Under ‘Reverse a string in Python’, most answers are based on reversing individual code points, including the naïve incumbent (+3007 / −5). The only answer pointing out that this is incorrect (+43 / −0) and offering an alternative based on grapheme clusters appears fourth in the naïve score sort and second under the ‘Best’ sort, behind the naïve incumbent. Again, the incumbent is probably not going to be displaced without more votes being cast against it, and in favour of the other answer. (This remains the case after I awarded a bounty to the latter answer.)
  • Under ‘What does "i" represent in Python .pyi extension?’, both naïve and weighted sort place the a speculative, meandering accepted answer (+139 / −2) first, and an evidence-based answer by @Pyprohly (+29 / −1) third.