How to use ruby to get string between html <cite> tags?

Question

Greetings everyone:

I would love to get some infomation from a huge collection of Google Search Result pages. The only thing I need is the urls inside a bunch of html tags.

I cannot get a solution in any other proper way to handle this problem so now I am moving to ruby.

This is so far what I have written:

require 'net/http'
require 'uri'

url=URI.parse('http://www.google.com.au')
res= Net::HTTP.start(url.host, url.port){|http|
    http.get('/#hl=en&q=helloworld')}
puts res.body

Unfortunately I cannot use the recommended hpricot ruby gem (because it misses a make command or something?)

So I would like to stick with this approach.

Now that I can get the response body as a string, the only thing I need is to retrive whatever is inside the ciite(remove an i to see the true name :)) html tags.

How should I do that? using regular expression? Can anyone give me an example?

Many thanks in advance!!!

Answer 1

Split the string on the tag you want. Assuming only one instance of tag (or specify only one split) you'll have two pieces I'll call head and tail. Take tail and split it on the closing tag (once), so you'll now have two pieces in your new array. The new head is what was between your tags, and the new tail is the remainder of the string, which you may process again if the tag could appear more than once.

An example that may not be exactly correct but you get the idea:

head1, tail1 = str.split('<tag>', 1) # finds the opening tag
head2, tail2 = tail1.split('</tag>', 1) # finds the closing tag

Answer 2

If you're having problems with hpricot, you could also try nokogiri which is very similar, and allows you to do the same things.

Answer 3

I think this will solve it:

res.scan(/<cite>([^<>]*)<\/cite>/imu).flatten

# This one to ignore empty tags:

res.scan(/<cite>([^<>]*)<\/cite>/imu).flatten.select{|x| !x.empty?}

Answer 4

Here's one way to do it using Nokogiri:

Nokogiri::HTML(res.body).css("cite").map {|cite| cite.content}