How to find duplicate rows in a join text file

Question

I have recently converted 10 JavaScript files into one file, which I then run a JavaScript compiler on. I just had a bug where I had reused a function name.

Is there a tool to check for duplicate rows/function names in the combined file?

Or should I create a little program?

Answer 1

I haven't tried this but the "Dupli Find" available at http://www.rlvision.com/dupli/about.asp may be of help to you.

The windows powershell script outlined in http://secretgeek.net/ps_duplicates.asp also helps you write a custom tool.

There is also a scripting solution at http://www.microsoft.com/technet/scriptcenter/resources/qanda/aug05/hey0819.mspx

Answer 2

cat file.js | grep -o "function\([[:space:]]\+[a-zA-Z0-9_]\+\)\?[[:space:]]*(" | sort | uniq -c | sort -n

Which reads:

• Cat file
• Look for function definitions (function-ws-name-ws-paren) the -o extract only matching parts of lines (ie only the definitions themselves)
• sort (for the next step)
• Count identical consequetive rows (uniq removes duplicates, -c adds a count)
• sort numerically by the occurrence count so that duplicates (if any appear last)

You could filter out non-duplicates but it's easier just to sort so they come first.

Edit Changed regex to include anonymous functions