Fastest / most efficient way to compare two string arrays Javascript

Question

Hi I was wondering whether anyone could offer some advice on the fastest / most efficient way to compre two arrays of strings in javascript.

I am developing a kind of tag cloud type thing based on a users input - the input being in the form a written piece of text such as a blog article or the likes.

I therefore have an array that I keep of words to not include - is, a, the etc etc.

At the moment i am doing the following:

Remove all punctuation from the input string, tokenize it, compare each word to the exclude array and then remove any duplicates.

The comparisons are preformed by looping over each item in the exclude array for every word in the input text - this seems kind of brute force and is crashing internet explorer on arrays of more than a few hundred words.

i should also mention my exclude list has around 300 items.

Any help would really be appreciated.

Thanks

Answer 1

You could use a hashing function for strings (I don't know if JS has one but i'm sure uncle Google can help ;] ). Then you would calculate hashes for all the words in your exclude list and create an array af booleans indexed by those hashes. Then just iterate through the text and check the word hashes against that array.

Answer 2

I'm not sure about the whole approach, but rather than building a huge array then iterating over it, why not put the "keys" into a map-"like" object for easier comparison?

e.g.

var excludes = {};//object
//set keys into the "map"
excludes['bad'] = true;
excludes['words'] = true;
excludes['exclude'] = true;
excludes['all'] = true;
excludes['these'] = true;

Then when you want to compare... just do

var wordsToTest = ['these','are','all','my','words','to','check','for'];
var checkWord;
for(var i=0;i<wordsToTest.length;i++){
  checkWord = wordsToTest[i];
  if(excludes[checkword]){
    //bad word, ignore...
  } else {
    //good word... do something with it
  }
}

allows these words through ['are','my','to','check','for']

Answer 3

It would be worth a try to combine the words into a single regex, and then compare with that. The regex engine's optimizations might allow the search to skip forward through the search text a lot more efficiently than you could do by iterating yourself over separate strings.