Convert a relative URL to an absolute URL with Simple HTML DOM?

Question

When I'm scraping content from some pages, the script gives a relative URL. Is it possible to get a absolute URL with Simple HTML DOM?

Answer 1

EDIT See Gumbo's answer for a formally correct answer. This is a simplified algorithm that will work in the vast majority of cases, but fail on some.

Sure. Do this:

• Take the relative URL (a URL that doesn't start with http://, https://, or any other protocol, and also doesn't start with /).
• Take the URL of the page.
• Remove the query string from it (if any). One simple way is to explode around ? and then take the first element of the resulting array (take element with index 0 or use reset).
  • If the URL of the page ends in /, append it the relative URL and you have the final URL.
  • If the URL doesn't end in /, take dirname of it, and append it the relative URL. You now have the final URL.

Answer 2

In addition to @Artefacto's answer, and if you are outputting the scraped HTML somewhere, you could simply add <base href="http://example.com"&gt; to the head of the document, which will establish the base URL for all relative URLs in the document as the specified href. Have a look at http://www.w3schools.com/tags/tag_base.asp

Answer 3

I don’t think that the Simple HTML DOM Parser can do that.

But you can do that on your own. First you need to distinguish the base URI that is the URI of the document if not declared otherwise (see BASE element). Than get each URI reference and apply the algorithms to resolve a relative URI as described in RFC 3986 (there already are classes you can use for that like the PEAR package Net_URL2).

So, using these two classes, you could do something like this:

$uri = new Net_URL2('http://example.com/foo/bar'); // URI of the resource
$baseURI = $uri;
foreach ($html->find('base[href]') as $elem) {
    $baseURI = $uri->resolve($elem->href);
}

foreach ($html->find('*[src]') as $elem) {
    $elem->src = $baseURI->resolve($elem->src)->__toString();
}
foreach ($html->find('*[href]') as $elem) {
    if (strtoupper($elem->tag) === 'BASE') continue;
    $elem->href = $baseURI->resolve($elem->href)->__toString();
}
foreach ($html->find('form[action]') as $elem) {
    $elem->action = $baseURI->resolve($elem->action)->__toString();
}

Repeat the substitution for any other attribute containing a URI like background, cite, classid, codebase, data, longdesc, profile and usemap (see index of attributes in HTML 4.01).