I want to parse a web page in Groovy and extract all of the href links and the associated text with it.
If the page contained these links:
<a href="http://www.google.com">Google</a>
<a href="http://www.apple.com">Apple</a>
The output would be:
Google, http://www.google.com
Apple, http://www.apple.com
I'm looking for a Groovy answer. AKA. The easy way!
A quick google search turned up a nice looking possibility, TagSoup.
depends which languages you know... In Java I use Apache common's HTTP Parser (along with their HTTPClient).
I'm sure that there is a widely used HTML parser for this in any language that you are developing in.
Try a regular expression. Something like this should work:
(html =~ /<a.*href='(.*?)'.*>(.*?)<\/a>/).each { url, text ->
// do something with url and text
}
Take a look at Groovy - Tutorial 4 - Regular expressions basics and Anchor Tag Regular Expression Breaking.
Use XMLSlurper to parse the HTML as an XML document and then use the find method with an appropriate closure to select the a tags and then use the list method on GPathResult to get a list of the tags. You should then be able to extract the text as children of the GPathResult.
I don't know java but I think that xpath is far better than classic regular expressions in order to get one (or more) html elements.
It is also easier to write and to read.
<html>
<body>
<a href="1.html">1</a>
<a href="2.html">2</a>
<a href="3.html">3</a>
</body>
</html>
With the html above, this expression "/html/body/a" will list all href elements.
Here's a good step by step tutorial http://www.zvon.org/xxl/XPathTutorial/General/examples.html
Assuming well-formed XHTML, slurp the xml, collect up all the tags, find the 'a' tags, and print out the href and text.
input = """<html><body>
<a href = "http://www.hjsoft.com/">John</a>
<a href = "http://www.google.com/">Google</a>
<a href = "http://www.stackoverflow.com/">StackOverflow</a>
</body></html>"""
doc = new XmlSlurper().parseText(input)
doc.depthFirst().collect { it }.findAll { it.name() == "a" }.each {
println "${it.text()}, ${[email protected]()}"
}
Html parser + Regular expressions Any language would do it, though I'd say Perl is the fastest solution.