Fast way to match an array of words with a block of text?

Question

Hi All,

The subject is probably not as clear as it could be, but I was struggling to think of a better way to easily describe it.

I am implementing a badword filter on some articles that we pick up from an XML feed. At the moment I have the badwords in an array and simply check the text like so;

str_replace($badwords, '', $text, $count); 
if ($count > 0) // We have bad words...

But this is SLOW! So slow! And when I am trying to process 30,000+ articles at a time, I start wondering if there is a better way to achieve this. If only strpos supported arrays! Even then I dont think it'd be faster...

I'd love any suggestions. Thanks in advance!

EDIT:

I have now tested a few methods between calls to microtime() to time them. str_replace() = 990 seconds preg_match() = 1029 seconds (Remember I only need to identify them, not replace them) no bad word filtering = 1057 seconds (presumably because it has another thousand or so bad-worded articles to process.

Thanks for all the answers, I will just still with str_replace. :)

Answer 1

How about combining all the words in a regex to replace everything in one go? I'm not sure how it will go for performance but it might be faster.

E.g.

preg_replace('/(' . implode('|', $badwords) . ')/i', '', $text);

Answer 2

Define "slow"? Anything that's going to be processing 30,000 articles is probably going to take a bit of time to complete.

That said, one option (which I have not benchmarked, just tossing it out there for consideration) would be to combine the words into a regex and run that through preg_replace (just using the | operator to put them together).

Answer 3

hi, i used to work at my local newspaper office. instead of modifying the text to delete badwords from the original files, what i did was just run a filter when a user requested to view the article. this way you preserve the original text should you ever need it, but also dish out a clean version for your viewers. there should be no need to process 30,000 articles at once unless i am misunderstanding something.

Answer 4

In case these previous questions are useful:

• How do you implement a good profanity filter?
• How do I replace bad words with php?
• Blacklist of words on content to filter message.
• Trouble with simple PHP profanity filter