Socially destructive and unhealthy communities can thrive on social networks, particularly on Reddit. Reddit is a platform that hosts pages that are defined by a topic/interest area where users can post and reply to other posts. These pages are called subreddits and the posts created on these pages we call threads. These topic areas are created by users, for users, and topics such as fat-shaming, white supremacy, sexual violence, etc., have taken root on Reddit. Reddit’s response to these communities is to restrict their activity, whether that be through temporary or permanent bans.
This project creates a model for Reddit’s banning patterns for the purpose of automatic detection of “bannable” content. I worked with three other people and we explored various algorithms, units of text, and transformations to determine the best model for detecting bannable subreddits.
Challenges
This project was very challenging, and one of the biggest barriers we faced was the scale of the data. Reddit produces so much data that we could not possibly utilize all of it. After doing some exploratory analysis, we determined that one month of data is a reasonable compromise between not enough data and too much data. The month we chose to use was October of 2016, and there was nearly 7 GB of comment data during this month. This scale of data was a size which our computers could handle, although performing complex analyses would still be difficult.
Working with textual data is very challenging, but it is even harder to work with short pieces of textual data. We originally tried models which were designed to predict whether a single comment was bannable or not. However, it quickly became apparent that we would need to use a different data representation, as individual comments were usually less than a sentence. After exploring many different ways of representing the data, we settled on combining all comments within a group into a single stream, and taking 200-word chunks from this stream. This ensures that each input example into any model we use has the same number of words, and therefore, roughly the same amount of information.
Some examples of comments which came from banned and not banned subreddits are shown in the image below. You can see that these comments are short, and generally not easy to classify. Some comments which appear in the banned category are innocent enough, and some comments from the not banned category have incel-related themes. This irregularity is another reason why we needed to create 200-word chunks: to add some noise to examples so that individual words did not have as much impact.
Another major issue with this project was the extreme imbalance of the data. Within the one month of data that we looked at, the ratio of not banned content to banned content was 183 to 1. Most methods and models are designed to work with balanced data, and they start to break down when the data becomes slightly imbalanced. This ratio, however, is so extreme that many models completely failed. They would group every single example into the majority class. We overcame this issue by utilizing a nonstandard loss function: the focal loss. This loss function, proposed by Lin et al. quite recently in 2017 is designed to train a neural network classifier on unbalanced data. It does this by placing a greater importance on classifying difficult-to-classify examples. The curves below show that depending on the parameter of gamma that you use, you can place even more importance on the difficult examples. For our purposes, we used a gamma value of 2.
Results
After spending quite a while on just getting models to be functional, we were able to achieve impressive results in the end. Shown below is the ROC curve for our model which shows, given a cutoff for splitting examples into banned and not banned groups, the precision and recall we can expect to get. The area under the ROC curve is 0.95 which is extremely high (highest possible value is 1). We ended up picking a fairly loose threshold which was designed to maximize recall. This way, we can overclassify subreddits as bannable, and then a human content moderation team can review these classifications and determine which subreddits should be banned. If recall was not high enough, they would miss too many at-risk subreddits for it to be worth it. Precision, however, is less critical, so we’re willing to take some sacrifices as long as our recall is increasing. On a test set of 15 now-banned subreddits, we were able to successfully recall 12 of them which is quite good. This indicates that, given more tuning, our model could be used to reduce the workload of Reddit content moderators.
The repository containing our paper, slides, and code is located here.
