Description
The aim of this project is to write a simple bot which will monitor all new answers posted on Stack Overflow and flag the ones which are not answers in accordance to the guidelines outlined in When should you decide whether to flag a posted answer as Not an answer? and How do I properly use the “Not an Answer” flag?
We already have Natty, why need another one?
The inspiration for this bot came from Natty - Bringing 10k moderation to All.
However, the goal of this bot is not to replace Natty neither is it to do the exact same thing. The goal is to supplement and flag answers which are not picked up by Natty.
Natty is an amazing bot and excels in finding flaggable content however, it purposefully avoids new questions, self-answers, and scans the content immediately after it has been posted. NAA posts on questions which are less than 30 days old are never caught by Natty, nor posts which were edited during the grace period.
Design criterion
- The bot should not repeat Natty's job. Posts flagged by Natty should be ignored by my NAABot
- It should be able to find NAA posts from the following categories: self-answers (intended edit), questions asked in answers, "me too" answers, "thanks" answers, gibberish, comments posted as answers and link-only answers
- The bot should identify problematic content as much as possible ignoring grammatical, typographical, and formatting errors
- It should have ~99% accuracy rate when auto-flagging
How was this achieved?
The bot uses a series of simple heuristics to calculate certainty score. Extensive research was conducted to analyse thousands of NAA and VLQ posts to identify characteristics indicating the post is not an answer and deserves deletion or converting to a comment. This research showed that there is no one single characteristic, but a combination of them is a good indicator of a flaggable material. Some words/phrases appear much more often in VLQ answers than they do in normal answers. Short answers, posts by low-reputation users or unregistered and posts ending with a question mark are very good indicators. However, it is not possible to apply a same weight to each heuristic. Some carry more weight than others, while some phrases make a post not an answer only in combination with multiple other heuristics.
Some of the heuristics applied are:
- Post ends in question mark or the question mark is one of the very last characters
- Contains some combination of "Thanks in Advance"
- Some variation of "Could you please help me fix this"
- Some variation of "Any help would be appreciated"
- Some variation of "Did you ever figure out"
- Some variation of "I have a similar problem"
- Lack of Latin characters
- More than half of the answer is made up of hyperlinks
- Low length (varying weight)
- Low reputation (varying weight)
Many of the blacklisted phrases checks are performed using regular expressions. This helps to catch the most common errors and variations of the same phrase.
To avoid flagging the same posts that Natty did, my bot checks Natty API first to see what score has Natty given to the post. If the post was not caught by Natty or Natty's score was too low for auto-flagging and my bot is certain then my bot will flag it instead and report to Natty. However, because the bot runs under my user and I can only flag 100 posts a day, some posts are not auto-flagged to avoid running out of the available flags. The assumption is that if it has been reported in SOBotics, someone will flag it manually.
Natty on some occasions didn't catch the post because the revision scanned by Natty was later edited during the grace period. To overcome this problem, my bot scans all new answers with a 5 minute delay. The assumption is that during the first 5 minutes OP can freely edit the post or delete it (if it was an accident). After that time the post should be scanned and if missed by Natty, reported in SOBotics.
If the problematic answer was posted on a new question then it will be flagged automatically and never reported in SOBotics.
Currently, the certainty threshold is set to 6. Any post which accrues a score of 6 or more is deemed to be delete-worthy and flaggable. It is acceptable that such post can simply be edited into shape by a more experienced user.
Anything with a score between 4-6 is a potentially flaggable. However, to achieve 99% accuracy in auto-flagging, these posts need to be manually reviewed.
Where can I find this bot?
The bot reports all potentially flaggable answers in a chat room: https://chat.stackoverflow.com/rooms/210133/naabot Besides the score, link, and short summary the bot also posts comments under some reports whether the post was auto-flagged, flagged by Natty, or left for manual attention.
A full audit log of the content scanned and heuristics applied to every post can be found at https://bot.dharman.net/reports.php
To see a list of all posts add a query string parameter: https://bot.dharman.net/reports.php?minScore=0
Does this bot require feedback?
No, I have no use for human feedback at the moment. This bot does not learn from past reports and the heuristics are fixed.
Can I help?
Sure, come on to the chat room where the reports are posted and help me to flag the answers. Due to the 100 flag a day limit, I am not able to flag everything. There are some posts which are yet waiting to be flagged.
