Regexp to match Javascript string literals with a specific keyword using Java

Question

I'm trying to match chunks of JS code and extract string literals that contain a given keyword using Java.

After trying to come up with my own regexp to do this, I ended up modifying this generalized string-literal matching regexp (Pattern.COMMENTS used when building the patterns in Java):

(["'])
(?:\\?+.)*?
\1

to the following

(["'])
(?:\\?+.)*?
keyword
(?:\\?+.)*?
\1

The test cases:

var v1 = "test";
var v2 = "testkeyword";
var v3 = "test"; var v4 = "testkeyword";

The regexp correctly doesn't match line 1 and correctly matches line 2.

However, in line 3, instead of just matching "testkeyword", it matches the chunk

"test"; var v4 = "testkeyword"

which is wrong - the regexp matched the first double quote and did not terminate at the second double quote, going all the way till the end of line.

Does anyone have any ideas on how to fix this?

PS: Please keep in mind that the Regexp has to correctly handle escaped single and double quote characters inside of string literals (which the generalized matcher already did).

Answer 1

After much revision (see edit history, viewers at home :), I believe this is my final answer:

(?:
    "
    (?:\\?+"|[^"])*
    keyword
    (?:\\?+"|[^"])*
    "
|
    '
    (?:\\?+'|[^'])*
    keyword
    (?:\\?+'|[^'])*
    '
)

Answer 2

How about this modification:

(?:
    "
    (?:\\"|[^"\r\n])*
    keyword
    (?:\\"|[^"\r\n])*
    "
|
    '
    (?:\\'|[^'\r\n])*
    keyword
    (?:\\'|[^'\r\n])*
    '
)

Answer 3

You need to write two patterns for either single or double quoted strings, as there is no way to make the regex remember which opened the string. Then you can or them together with |.

Answer 4

Consider using code from Rhino -- JS in Java -- to get the real String literals.

Or, if you want to use regex, consider one find for the whole literal, then a nested test if the literal contains 'keyword'.

I think Tim's construction works, but I wouldn't bet on it in all situations, and the regex would have to get insanely unwieldy if it had to deal with literals that don't want to be found (as if trying to sneak by your testing). For example:

    var v5 =  "test\x6b\u0065yword"

Separate from any solution, my secret weapon for interactively working out regexes is a tool I made called Regex Powertoy, which unlike many such utilities runs in any browser with Java applet support.

Answer 5

A grammar to construct a string literal would look roughly like this:

string-literal ::= quote text quote

text ::= character text
       | character

character ::= non-quote
            | backslash quote

with non-quote, backslash, and quote being terminals.

A grammar is regular if it is context free (i.e. the left hand side of all rules is always a single non-terminal) and the right hand side of all rules is always either empty, a terminal, or a terminal followed by a non-terminal.

You may notice that the first rule given above has a terminal followed by a nonterminal followed by a terminal. This is thus not a regular grammar.

A regular expression is an expression that can parse regular languages (languages that can be constructed by a regular grammar). It is not possible to parse non-regular languages with regular expressions.

The difficulty you have in finding a suitable regular expression stems from the fact that a suitable regular expression doesn't exist. You will never arrive at code that is obviously correct, this way.

It is much easier to write a simple parser along the lines of above rules. Since the text contained by your string literals is regular, you can use a simple regular expression to look for your keyword---after you extracted that text from its surroundings.