Ruby Regex Help

Question

Hi im learning Ruby Regex, but having once problem here.

I want to extract links from google.com Code i shall extract from looks like this

<a href="http://www.test.com/" class="l"

I took me around 5min to found a regex that shall works. (with www.rubular.com) It is

"(.*?)" class="l"

Heres Comes the code

require "open-uri"
url = "http://www.google.com/search?q=ruby"

source = open(url).read()
links = source.scan(/"(.*?)" class="l"/) 

links.each {|link| puts #{link} 

}

Problem is it, is it the program dont outputs the websites links. What im going wrong?

Answer 1

What im going wrong?

You're trying to parse HTML with regex. Don't do that. Regular expressions cannot cover the range of syntax allowed even by valid XHTML, let alone real-world tag soup. Use an HTML parser library such as Hpricot.

FWIW, when I fetch ‘http://www.google.com/search?q=ruby’ I do not receive ‘class="l"’ anywhere in the returned markup. Perhaps it depends on which local Google you are using and/or whether you are logged in or otherwise have a Google cookie. (Your script, like me, would not.)

Answer 2

Those links actually have class=l not class="l". By the way, to figure this put I added some logging to the method so that you can see the output at various stages and debug it. I searched for the string you were expecting to find and didn't find it, which is why your regex failed. So I looked for the right string you actually wanted and changed the regex accordingly. Debugging skills are handy.

require "open-uri"
url = "http://www.google.com/search?q=ruby"

source = open(url).read

puts "--- PAGE SOURCE ---"
puts source

links = source.scan(/<a.+?href="(.+?)".+?class=l/)

puts "--- FOUND THIS MANY LINKS ---"
puts links.size

puts "--- PRINTING LINKS ---"
links.each do |link|
  puts "- #{link}"
end

I also improved your regex. You are looking for some text that starts with the opening of an a tag (<a), then some characters of some sort that you dont care about (.+?), an href attribute (href="), the contents of the href attribute that you want to capture ((.+?)), some spaces or other attributes (.+?), and lastly the class attrubute (class=l).

I have .+? in three places there. the . means any character, the + means there must be one or more of the things right before it, and the ? means that the .+ should try to match as short a string as possible.

Answer 3

To put it bluntly, the problem is that you're using regexes. The problem is that HTML is what is known as a context-free language, while regular expressions can only the class of languages that are known as regular languages.

What you should do is send the page data to a parser that can handle HTML code, such as Hpricot, and then walk the parse tree you get from the parser.