How are training comments chosen? How are they classified?
Training comments are taken with a simple API request which will request the newest page of comments. This is done to make the training comments exactly like what the bot would be classifying while in use. The comments are then manually labeled by me, as "useful" (not obviously NLN without context) or "not useful" (clearly NLN without context).
How does the model's training and evaluation work?
The BERT-BASE model is fine-tuned on the comments for 3 epochs on my CPU. When training a model for use, I'll use all the comments for training. When training a model for testing, I use train_test_split() with test_size = 0.3. I use a standard learning_rate of 2e-5. per_device_train_batch_size, per_device_eval_batch_size, and gradient_accumulation_steps are all set to 1. For the loss function, I use PyTorch's torch.nn.CrossEntropyLoss(). I have also been experimenting with a new loss function, which is a combination of the previous function and another loss function, which rewards and penalizes the model more for greater confidence exponentially. This could be used to get more realistic accuracy confidence from the bot, which I am currently not sending along with chat messages because of how misleading it is (the model is more than 99.99999% about every comment, apparently). The comments are then tokenized with tokenizer(texts, padding="max_length", truncation=True, max_length=488) (the max_length is set to 488 because that is the largest length of any of the comments in the dataset, I have another program which finds this).
If training it to test how well the model works (using a test_size = 0.3 as mentioned above), the model will then evaluate itself on the 30% of comments reserved for testing, and will return eval_loss (the loss on the test comments), eval_overall_accuracy (overall accuracy), eval_useful_accuracy (accuracy on comments which were actually useful), and eval_not_useful_accuracy (accuracy on comments which were actually not useful).
Aren't there issues with class imbalance?
Well, yes. As mentioned above, the comments chosen are a random sampling of comments on Stack Overflow. Thankfully, most comments are useful, which results in a pretty heavy class imbalance (about 12.8 useful comments for 1 not useful one). To work around this, I've used sklearn's compute_class_weight(), setting class_weight='balanced', to automatically compute class weights. Currently, the computed class weights are around 6.86 for not useful and 0.54 for useful.
The class imbalance has 2 side affects as well. First, below about 800 training comments or so, the model is virtually useless. Even with compute_class_weight(), the model will always predict useful. Even setting insane manual class weights (ex. 100 for not useful and 0.1 for useful) still makes no difference. Second, the model is significantly better at predicting useful comments correctly. This turns out to be quite useful, as the 12.8 to 1 imbalance would otherwise cause a huge rate of false positives. For example if 80% of useful and 80% of not useful comments were predicted correctly, the model would actually have a false positive (comment predicted as not useful when it is actually useful) rate of about 76.2%, which would make the bot quite annoying, reporting a huge number of fine comments.
How accurate is the model?
(I understand that parts of the last 2 sections were pretty technical. If you didn't understand them, it's fine, you will be able to understand this section without understanding the past 2.)
So, with the 4,000 comments I labeled (technically 4001), I split 30% off for testing (so 1201 comments), and left 70% for training (so 2800 comments). After training on the training comments, I got back the results for the model's accuracy.
The model's eval_overall_accuracy (accuracy on all comments) was 96.59%. The model's eval_useful_accuracy (% of the time when a comment that was actually useful was predicted correctly) was 99.37%. The model's eval_not_useful_accuracy (% of the time when a comment that was actually not useful was predicted correctly) was 60.47%.
So, what would be the false positive rate for the bot (% of the time when a comment predicted as not useful is actually useful)? Taking into account the class imbalance from before (basically, that there's a lot more useful comments than not useful comments), the expected false positive rate would be around 11.8%. Not terrible.
Does the false positive rate actually correlate with real life performance?
Well, most of the time, yes. The bot generally works pretty quite well and roughly how I would expect it to work from testing and training it. There is one exception to this though.
Many comments contain an @ reply to some user, which is shown in the comment as "@some user" (some user is used a placeholder username, this applies to all usernames, it seems like). The model is then trained on such a comment, and since that comment happened to be not useful, the model thinks it has learned that comments containing "@some user" are not useful. This then results in the model reporting almost every comment containing "@some user", and sometimes even just a comment containing "some user". This could also occur in reverse, where the comment containing "@some user" is classified as useful, so it doesn't report any comments with "@some user" or "some user".
Fortunately, this issue seems to be relatively rare, but it does exist.
Does this bot autoflag comments? Are there plans for it to do so in the future?
No, this bot doesn't autoflag comments currently. The main issue here is accuracy. As mentioned above, the bot has a false positive rate of 11.8%, so it has an accuracy of 88.2%, which is fine for reporting.
For flagging, though, this is too low accuracy. I would like to see 99% or greater accuracy before I would consider auto-flagging comments.
