How can I moderate trolls algorithmically?

Question

I'm writing a forum application but one of the things I'm concerned about is trolls - users who disrupt dialog with abusive speech or off-topic content. This goes beyond spam prevention because it includes people who are actually participating in a discussion but who either refuse or are unable to follow proper standards of behavior.

Without an ability to moderate these types of users algorithmically (manual moderation would require too much time and attention), I don't see how the forum can be successful since their participation is not only disruptive but also discourages the types of users I am interested in, from participating.

Are there specific features I can add to my app that would minimize the disruptive effect of trolls while also minimizing the barriers to entry for new users?

The only feature I can think of that would not require as much active user moderation is identity verification - such as a cell phone number - which I would then actually have to verify belongs to this person. But this creates a significant barrier to entry.

Having a "flag this" link next to user content seems itself to be prone to abuse and require moderation.

Answer 1

You can't determine trolling algorithmically without Star Trek-like universal language parsers.

But you can do the next best thing, which is what Stack Overflow does: give various moderation rights to every user. Each user does a little bit of moderation so the primary moderators don't have to put in as much effort.

After very little time, users on SO are able to flag any post as offensive for moderators to see. After more time, they can edit posts, tags, close questions and even become mini moderators themselves. On SO the process of getting these rights is procedural, but the moderation itself is not. That's a good approach.

In lower-tech applications like simple forums or IRC channels, simply spreading the load is usually good enough. Instead of having a few moderators with all of the rights, you give regular, trustworthy users (of which there are probably many) a subset of rights to help with moderation. It works very well, since most active community members will do this kind of thing for free, just to see a better community.

Answer 2

The something awful forums have a $10 fee that can be only paid for by credit card. If you screw up you can be put on probation (lose posting access for X amount of time) or banned (have to pay another $10).

Of course, your users may not be willing to pay money, but it does help to keep away children (unless their parents are willing to give credit card access) and from trolls that don't really care about what they're doing.

Answer 3

Your goal should be to use algorithms and automated processes to reduce the number of posts that your moderators must manually review. By running processes that check for posts containing hate speech or no text related to the original post (both very non trivial algorithms) you can reduce the workload for your moderators. It is also possible to promote your best community members to moderators to reduce the per person work load moderating the forum.

Answer 4

At the very least, you can have a list of offensive words/phrases and create a script that runs when a user with less than X posts, or has been registered for less than Y days makes a post. If one or more of the offensive words/phrases is found, the script could send you a "possible troll" email.

Answer 5

A good way to control such abuse is to eliminate, or reduce as much as possible, anonymity. If you have a scenario where people know their words can be traced back to them, you'd be surprised how much less likely they are to misbehave.

This article on ReadWriteWeb tackles the topic very well

Answer 6

I would suggest a clever way of flagging replies, but be very specific.

For example, registered users can flag a post/reply using various specific types such as:

• Personal attack, ad hominem
• Off topic
• Spam

You could then provide a listing for admins to view the most flagged replies.

Answer 7

Probably it's just a brainstorming idea.... you can collect many troll's post, perform a frequencies analysys on using of some particular words, emoticons, exclamation marks ecc ecc, then you can find a threshold level for each of this particular items and develop an algorithm to ananlize each post.

I reapeat, it's just a fool idea!

enother one, to avoid, for example, randomly typed words, is entropy, you can calculate the entropy of the post.

Answer 8

Flags are a good way to go, do not underestimate the power of user feedback, whether that be by reputation, or by reporting offensive material to be reviewed by a mod.

Algorithms are going to be an awkward way forward, the star trek language parser is a little unrealistic but you could have a couple of dictionaries and match words contained in each dictionary separately that would say for example match a word like "anorexia" in one instance and then guess if its being used in a positive context by the surrounding words then flag it as questionable (unless you want pro-ana content). You would have to have dictionaries of words and phrases suggesting a positive sense, words and phrases that suggest a negative sense, words sensitive in positive context, words sensitive in negative context, then match those against the thread of the conversation average use of those terms being positive/negative so that posts going in the opposite direction of the thread could be flagged on the basis of disruptive to the flow of conversation if its sensitive subject areas then content to be reviewed by an admin. Pretty wishy washy, and only a small step from blind censorship but its about all i can think of (in my tiny incapable mind lol).

Answer 9

Some people are trying to develop filters for stupidity or trolling. For example, Stupid Filter claims to be "an open-source filter software that can detect rampant stupidity in written English."