process all links but external ones (ruby + mechanize)

Question

I want to process all links but external ones from the whole web site. Is there any easy way how to identify that the link is external and skip it?

My code looks so far like (the site url is passed through command line argument)

I am using mechanize (0.9.3) and ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]

Please note that the web site can use relative path so there is no host/domain and it makes it bit more complicated

require 'mechanize'

def process_page(page) 
  puts
  puts page.title
  STDIN.gets
  page.links.each do |link|
process_page($agent.get(link.href))
  end
end

$agent = WWW::Mechanize.new 
$agent.user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4'
process_page($agent.get(ARGV[0]))

Answer 1

Use the link's uri method:

  page.links.each do |link|
     next unless link.uri.host.match(/(www\.)?thissite\.com/)
     process_page($agent.get(link.href))
  end

Answer 2

URI has some methods that make it pretty easy to see whether you are looking at a local URL or one on another site.

This is a minor modification from the URI .route_to() docs example:

require 'uri'

URI.parse('/main.rbx?page=1').host # => nil
URI.parse('main.rbx?page=1').host  # => nil

Internal URLs have no host so I'd parse the URLs in question and look to see if they have a host. If not, it's internal to the site.

A URL pointing to an external site will return a value for the host but so will a full URL for the site in question so you have to do some more massaging.

uri = URI.parse('http://my.example.com')

uri.route_to('http://my.example.com/main.rbx?page=1').host  # => nil
uri.route_to('http://another.com/main.rbx?page=1').host # => "another.com"

If it has a host see whether that host matches your starting URL's host. You can do that by a substring search or a regex match, but both of those have a possibility of returning false-positives if a sub-string match occurs.

Instead, I'd use URI's methods to avoid those false positives; Use route_to() to try to build a relative path to the URL. If the result has a .host value then it's external.