Question about parsing HTML using Regex and Java

Question

I Have a question about finding html tags using Java and Regex.

I am using the code below to find all the tags in HTML, documentURL is obviously the HTML content.

The find method return true, meaning that it can find something in the HTML but the matches() method always return false and I am completly and utterly puzzled about this.

I refered to Java documentations too but could not find my answer.

What is the correct way of using Matcher ?

    Pattern keyLineContents = Pattern.compile("(<.*?>)");

    Matcher keyLineMatcher = keyLineContents.matcher(documentURL);

    boolean result = keyLineMatcher.find();

    boolean matchFound = keyLineMatcher.matches();

Doing something like this throws an exeption:

     String abc = keyLineMatcher.group(0);

Thanks.

Answer 1

The correct way to loop through matches is:

Pattern p = Pattern.compile("<.*?>");
Matcher m = p.matcher(htmlString);
while (m.find()) {
  System.out.println(m.group());
}

That being said, regular expressions are an extremely poor method of parsing HTML. The reason comes down to this: regular expressions work well for parsing regular languages. HTML is a context free language. Where regular expressions fall down is for things like nested tags, using > inside attribute values and so on.

Use a dedicated HTML parser instead such as HTML Parser.

Answer 2

What if someone wants to write his/her own HTML parser.

If not Regex then, What is the best way to do it ?

Answer 3

Why don't you try looking at the source code of some open source HTML Parsers? HtmlCleaner, Tagsoup etc.

The general strategy seems to be to attempt to parse and clean the html and return an Xml tree.

Personally, I would read through the HTML adding opening tags to a LIFO Queue, and removing (matching) opening tags from the start of the queue when a closing tag is encountered - performing queue shifting to allow for tag mismatches.

Answer 4

I want to get keyword content from HTML tag I wrote:

Pattern keyLineContents = Pattern.compile("<(.[^<]*)(keywords)(.[^<]*)>");
Matcher keyLineMatcher = keyLineContents.matcher(documentURL);
boolean result = keyLineMatcher.find();
while(result)
{
  String metaTagContent = keyLineMatcher.group(1) + " " + keyLineMatcher.group(3);
  Pattern kcontent = Pattern.compile("(.*?content=\")(.[^<]*?)(\".[^<]*?)");
  Matcher keyLineMatcher2 = kcontent.matcher(metaTagContent);
  boolean result2 = keyLineMatcher.find();
  while (result2)
  {
    String metaTagContent2 = keyLineMatcher.group(1);
    result2 = keyLineMatcher.find();
  }
}

But I don't understand why my result2 is false. Result one is fine give all content of keyword tag

thanks