How to match a comment unless it's in a quoted string?

Question

So I have some string:

//Blah blah blach
// sdfkjlasdf
"Another //thing"

And I'm using java regex to replace all the lines that have double slashes like so:

theString = Pattern.compile("//(.*?)\\n", Pattern.DOTALL).matcher(theString).replaceAll("");

And it works for the most part, but the problem is it removes all the occurrences and I need to find a way to have it not remove the quoted occurrence. How would I go about doing that?

Answer 1

Use a parser, determine it char-by-char.

Kickoff example:

StringBuilder builder = new StringBuilder();
boolean quoted = false;

for (String line : string.split("\\n")) {
    for (int i = 0; i < line.length(); i++) {
        char c = line.charAt(i);
        if (c == '"') {
            quoted = !quoted;
        }
        if (!quoted && c == '/' && i + 1 < line.length() && line.charAt(i + 1) == '/') {
            break;
        } else {
            builder.append(c);
        }
    }
    builder.append("\n");
}

String parsed = builder.toString();
System.out.println(parsed);

Answer 2

You can't tell using regex if you are in double quoted string or not. In the end regex is just a state machine (sometimes extended abit). I would use a parser as provided by BalusC or this one.

If you want know why the regex are limited read about formal grammars. A wikipedia article is a good start.

Answer 3

Instead of using a parser that parses an entire Java source file, or writing something yourself that parses only those parts you're interested in, you could use some 3rd party tool like ANTLR.

ANTLR has the ability to define only those tokens you are interested in (and of course the tokens that can mess up your token-stream like multi-line comments and String- and char literals). So you only need to define a lexer (another word for tokenizer) that correctly handles those tokens.

This is called a grammar. In ANTLR, such a grammar could look like this:

lexer grammar FuzzyJavaLexer;

options{filter=true;}

SingleLineComment
  :  '//' ~( '\r' | '\n' )*
  ;

MultiLineComment
  :  '/*' .* '*/'
  ;

StringLiteral
  :  '"' ( '\\' . | ~( '"' | '\\' ) )* '"'
  ;

CharLiteral
  :  '\'' ( '\\' . | ~( '\'' | '\\' ) )* '\''
  ;

Save the above in a file called FuzzyJavaLexer.g. Now download ANTLR 3.2 here and save it in the same folder as your FuzzyJavaLexer.g file.

Execute the following command:

java -cp antlr-3.2.jar org.antlr.Tool FuzzyJavaLexer.g

which will create a FuzzyJavaLexer.java source class.

Of course you need to test the lexer, which you can do by creating a file called FuzzyJavaLexerTest.java and copying the code below in it:

import org.antlr.runtime.*;

public class FuzzyJavaLexerTest {
    public static void main(String[] args) throws Exception {
        String source = 
            "class Test {                                 \n"+
            "  String s = \" ... \\\" // no comment \";   \n"+
            "  /*                                         \n"+
            "   * also no comment: // foo                 \n"+
            "   */                                        \n"+
            "  char quote = '\"';                         \n"+
            "  // yes, a comment, finally!!!              \n"+
            "  int i = 0; // another comment              \n"+
            "}                                            \n";
        System.out.println("===== source =====");
        System.out.println(source);
        System.out.println("==================");
        ANTLRStringStream in = new ANTLRStringStream(source);
        FuzzyJavaLexer lexer = new FuzzyJavaLexer(in);
        CommonTokenStream tokens = new CommonTokenStream(lexer);
        for(Object obj : tokens.getTokens()) {
            Token token = (Token)obj;
            if(token.getType() == FuzzyJavaLexer.SingleLineComment) {
                System.out.println("Found a SingleLineComment on line "+token.getLine()+
                        ", starting at column "+token.getCharPositionInLine()+
                        ", text: "+token.getText());
            }
        }
    }
}

Next, compile your FuzzyJavaLexer.java and FuzzyJavaLexerTest.java by doing:

javac -cp .:antlr-3.2.jar *.java

and finally execute the FuzzyJavaLexerTest.class file:

// *nix/MacOS
java -cp .:antlr-3.2.jar FuzzyJavaLexerTest

or:

// Windows
java -cp .;antlr-3.2.jar FuzzyJavaLexerTest

after which you'll see the following being printed to your console:

===== source =====
class Test {                                 
  String s = " ... \" // no comment ";   
  /*                                         
   * also no comment: // foo                 
   */                                        
  char quote = '"';                         
  // yes, a comment, finally!!!              
  int i = 0; // another comment              
}                                            

==================
Found a SingleLineComment on line 7, starting at column 2, text: // yes, a comment, finally!!!              
Found a SingleLineComment on line 8, starting at column 13, text: // another comment  

Pretty easy, eh? :)

Answer 4

(This is in answer to the question @finnw asked in the comment under his answer. It's not so much an answer to the OP's question as an extended explanation of why a regex is the wrong tool.)

Here's my test code:

String r0 = "(?m)^((?:[^\"]|\"(?:[^\"]|\\\")*\")*)//.*$";
String r1 = "(?m)^((?:[^\"\r\n]|\"(?:[^\"\r\n]|\\\")*\")*)//.*$";
String r2 = "(?m)^((?:[^\"\r\n]|\"(?:[^\"\r\n\\\\]|\\\\\")*\")*)//.*$";

String test = 
    "class Test {                                 \n"+
    "  String s = \" ... \\\" // no comment \";   \n"+
    "  /*                                         \n"+
    "   * also no comment: // but no harm         \n"+
    "   */                                        \n"+
    "  /* no comment: // much harm  */            \n"+
    "  char quote = '\"';  // comment             \n"+
    "  // another comment                         \n"+
    "  int i = 0; // and another                  \n"+
    "}                                            \n"
    .replaceAll(" +$", "");
System.out.printf("%n%s%n", test);

System.out.printf("%n%s%n", test.replaceAll(r0, "$1"));
System.out.printf("%n%s%n", test.replaceAll(r1, "$1"));
System.out.printf("%n%s%n", test.replaceAll(r2, "$1"));

r0 is the edited regex from your answer; it removes only the final comment (// and another), because everything else is matched in group(1). Setting multiline mode ((?m)) is necessary for ^ and $ to work right, but it doesn't solve this problem because your character classes can still match newlines.

r1 deals with the newline problem, but it still incorrectly matches // no comment in the string literal, for two reasons: you didn't include a backslash in the first part of (?:[^\"\r\n]|\\\"); and you only used two of them to match the backslash in the second part.

r2 fixes that, but it makes no attempt to deal with the quote in the char literal, or single-line comments inside the multiline comments. They can probably be handled too, but this regex is already Baby Godzilla; do you really want to see it all grown up?.

Answer 5

The following is from a grep-like program I wrote (in Perl) a few years ago. It has an option to strip java comments before processing the file:

# ============================================================================
# ============================================================================
#
# strip_java_comments
# -------------------
#
# Strip the comments from a Java-like file.  Multi-line comments are
# replaced with the equivalent number of blank lines so that all text
# left behind stays on the same line.
#
# Comments are replaced by at least one space .
#
# The text for an entire file is assumed to be in $_ and is returned
# in $_
#
# ============================================================================
# ============================================================================

sub strip_java_comments
{
      s!(  (?: \" [^\"\\]*   (?:  \\.  [^\"\\]* )*  \" )
         | (?: \' [^\'\\]*   (?:  \\.  [^\'\\]* )*  \' )
         | (?: \/\/  [^\n] *)
         | (?: \/\*  .*? \*\/)
       )
       !
         my $x = $1;
         my $first = substr($x, 0, 1);
         if ($first eq '/')
         {
             "\n" x ($x =~ tr/\n//);
         }
         else
         {
             $x;
         }
       !esxg;
}

This code does actually work properly and can't be fooled by tricky comment/quote combinations. It will probably be fooled by unicode escapes (\u0022 etc), but you can easily deal with those first if you want to.

As it's Perl, not java, the replacement code will have to change. I'll have a quick crack at producing equivalent java. Stand by...

EDIT: I've just whipped this up. Will probably need work:

// The trick is to search for both comments and quoted strings.
// That way we won't notice a (partial or full) comment withing a quoted string
// or a (partial or full) quoted-string within a comment.
// (I may not have translated the back-slashes accurately.  You'll figure it out)

Pattern p = Pattern.compile(
       "(  (?: \" [^\"\\\\]*   (?:  \\\\.  [^\"\\\\]* )*  \" )" +  //    " ... "
       "  | (?: ' [^'\\\\]*    (?:  \\\\.  [^'\\\\]*  )*  '  )" +  // or ' ... '
       "  | (?: //  [^\\n] *    )" +                               // or // ...
       "  | (?: /\\*  .*? \\* / )" +                               // or /* ... */
       ")",
       Pattern.DOTALL  | Pattern.COMMENTS
);

Matcher m = p.matcher(entireInputFileAsAString);

StringBuilder output = new StringBuilder();

while (m.find())
{
    if (m.group(1).startsWith("/"))
    {
        // This is a comment. Replace it with a space...
        m.appendReplacement(output, " ");

        // ... or replace it with an equivalent number of newlines
        // (exercise for reader)
    }
    else
    {
        // We matched a quoted string.  Put it back
        m.appendReplacement(output, "$1");
    }
}

m.appendTail(output);
return output.toString();