Web scraping with Java

Question

I'm not able to find any good web scraping Java based API. The site which I need to scrape does not provide any API as well; I want to iterate over all web pages using some pageID and extract the HTML titles / other stuff in their DOM trees.

Are there ways other than web scraping?

Thanks

Answer 1

extract title is not difficult, and you have many options ( search here in SO for HTML Parsers in Java )

one of them is use: http://jsoup.org/

you can navigate the page using DOM if you know the page structure: http://jsoup.org/cookbook/extracting-data/dom-navigation

its a good library and i use it in my last projects.

Answer 2

I would suggest you a combination of Groovy and HtmlUnit. For lower level handling, you can use HttpBuilder.

Answer 3

Look at an HTML parser such as TagSoup, HTMLCleaner or NekoHTML.

Answer 4

Thanks! JSoup was what I was looking for.

Answer 5

Your best bet is to use Selenium Web Driver since it

1. Provides visual feedback to the coder (see your scraping in action, see where it stops)
2. Accurate and Consistent as it directly controls the browser you use.

3. Slow. Doesn't hit web pages like HtmlUnit does but sometimes you don't want to hit too fast.

   Htmlunit is fast but is horrible at handling Javascript and AJAX.