Natural Language Processing: Find obscenities in English?

Question

Given a set of words tagged for part of speech, I want to find those that are obscenities in mainstream English. How might I do this? Should I just make a huge list, and check for the presence of anything in the list? Should I try to use a regex to capture a bunch of variations on a single root?

If it makes it easier, I don't want to filter out, just to get a count. So if there are some false positives, it's not the end of the world, as long as there's a more or less uniformly over exaggerated rate.

Answer 1

I'd make a huge list.

Regex'es have the problem of misfiring, when applied to natural language - especially with an amount of exceptions English has.

Answer 2

It depends what your text source is, but I'd go for some kind of established and proven pattern matching algorithm, using a Trie for example.

Answer 3

Note that any NLP logic like this will be subject to attacks of "character replacement":

For example, I can write "hello" as "he11o", replacing L's with One's. Same with obscenities. So while there's no perfect answer, a "blacklist" approach of "bad words" might work. Watch out for false positives (I'd run my blacklist against a large book to see what comes up)

Answer 4

Use the morphy lemmatizer built into WordNet, and then determine whether the lemma is an obscenity. This will solve the problem of different verb forms, plurals, etc...

Answer 5

A huge list and think of the target audience. Is there 3rd party service that you can use that specialises in this rather than rolling your own?

Some quick thoughts:

• The Scunthorpe problem (and follow the links to "Swear filter" for more)
• British or American English? fanny, fag etc
• Political correctness: "black" or "Afro-American"?

Edit:

• Be very careful

Answer 6

One problem with filters of this kind is their tendency to flag entirely proper English town names like Scunthorpe. While that can be reduced by checking the whole word rather than parts, you then find people taking advantage by merging their offensive words with adjacent text.

Answer 7

I would advocate a large list of simple regex's. Smaller than a list of the variants, but not trying to capture anything more than letter alternatives in any given expression: like "f[u_-@#$%^&*.]ck".

Answer 8

You want to use Bayesian Analysis to solve this problem. Bayesian probability is a powerful technique used by spam filters to detect spam/phishing messages in your email inbox. You can train your analysis engine so that it can improve over time. The ability to detect a legitimate email vs. a spam email sounds identical to the problem you are experiencing.

Here are a couple of useful links:

A Plan For Spam - The first proposal to use Bayesian analysis to combat spam.

Data Mining (ppt) - This was written by a colleague of mine.

Classifier4J - A text classifier library written in Java (they exist for every language, but you tagged this question with Java).

Answer 9

Is the phrase I want to stick my long-necked Giraffe up your fluffy white bunny obscene?

Answer 10

There are webservices that do this kind of thing in English.

I'm sure there are others, but I've used WebPurify in a project for precisely this reason before.