Text Extraction from HTML Java

Question

Hi.

I'm working on a program that downloads HTML pages and then selects some of the information and write it to another file.

I want to extract the information which is intbetween the paragraph tags, but i can only get one line of the paragraph. My code is as follows;

FileReader fileReader = new FileReader(file);
 BufferedReader buffRd = new BufferedReader(fileReader);
 BufferedWriter out = new BufferedWriter(new FileWriter(newFile.txt));
 String s;
 while ((s = br.readLine()) !=null) {
  if(s.contains("<p>")){

   try {
          out.write(s);
      } catch (IOException e) {
      }
  }

i was trying to add another while loop, which would tell the program to keep writing to file until the line contains the </p> tag, by saying;

while ((s = br.readLine()) !=null) {
  if(s.contains("<p>")){
                    while(!s.contains("</p>"){
   try {
          out.write(s);
      } catch (IOException e) {
  }

But this doesn't work. Could someone please help.

Answer 1

Why don't you use an HTML parser for this?

Answer 2

Try (if you don't want to use a HTML parser library):

        FileReader fileReader = new FileReader(file);
        BufferedReader buffRd = new BufferedReader(fileReader);
        BufferedWriter out = new BufferedWriter(new FileWriter(newFile.txt));
        String s;
        int writeTo = 0;
        while ((s = br.readLine()) !=null) 
        {
                if(s.contains("<p>"))
             {
                  writeTo = 1;

                  try 
                  {
                      out.write(s);
                 } 
                  catch (IOException e) 
                  {

                 }
             }
             if(s.contains("</p>"))
             {
                  writeTo = 0;

                  try 
                  {
                      out.write(s);
                 } 
                  catch (IOException e) 
                  {

                 }
             }
             else if(writeTo==1)
             {
                  try 
                  {
                      out.write(s);
                 } 
                  catch (IOException e) 
                  {

                 }
             }
}

Answer 3

jericho is one of several posible html parsers that could make this task both easy and safe.

Answer 4

JTidy can represent an HTML document (even a malformed one) as a document model, making the process of extracting the contents of a <p> tag a rather more elegant process than manually thunking through the raw text.

Answer 5

You may just be using the wrong tool for the job:

perl -ne "print if m|<p>| .. m|</p>|" infile.txt >outfile.txt

Answer 6

I've had success using TagSoup & XPath to parse HTML.

http://home.ccil.org/~cowan/XML/tagsoup/

Answer 7

Use a ParserCallback. Its a simple class thats included with the JDK. It notifies you every time a new tag is found and then you can extract the text of the tag. Simple example:

import java.io.*;
import java.net.*;
import javax.swing.text.*;
import javax.swing.text.html.*;
import javax.swing.text.html.parser.*;

public class ParserCallbackTest extends HTMLEditorKit.ParserCallback
{
    private int tabLevel = 1;
    private int line = 1;

    public void handleComment(char[] data, int pos)
    {
     displayData(new String(data));
    }

    public void handleEndOfLineString(String eol)
    {
     System.out.println( line++ );
    }

    public void handleEndTag(HTML.Tag tag, int pos)
    {
     tabLevel--;
     displayData("/" + tag);
    }

    public void handleError(String errorMsg, int pos)
    {
     displayData(pos + ":" + errorMsg);
    }

    public void handleMutableTag(HTML.Tag tag, MutableAttributeSet a, int pos)
    {
     displayData("mutable:" + tag + ": " + pos + ": " + a);
    }

    public void handleSimpleTag(HTML.Tag tag, MutableAttributeSet a, int pos)
    {
     displayData( tag + "::" + a );
//   tabLevel++;
    }

    public void handleStartTag(HTML.Tag tag, MutableAttributeSet a, int pos)
    {
     displayData( tag + ":" + a );
     tabLevel++;
    }

    public void handleText(char[] data, int pos)
    {
     displayData( new String(data) );
    }

    private void displayData(String text)
    {
     for (int i = 0; i < tabLevel; i++)
      System.out.print("\t");

     System.out.println(text);
    }

    public static void main(String[] args)
    throws IOException
    {
     ParserCallbackTest parser = new ParserCallbackTest();

     // args[0] is the file to parse

     Reader reader = new FileReader(args[0]);
//   URLConnection conn = new URL(args[0]).openConnection();
//   Reader reader = new InputStreamReader(conn.getInputStream());

     try
     {
      new ParserDelegator().parse(reader, parser, true);
     }
     catch (IOException e)
     {
      System.out.println(e);
     }
    }
}

So all you need to do is set a boolean flag when the paragraph tag is found. Then in the handleText() method you extract the text.