C# Convert Relative to Absolute Links in HTML String

Question

I'm mirroring some internal websites for backup purposes. As of right now I basically use this c# code:

System.Net.WebClient client = new System.Net.WebClient();
byte[] dl = client.DownloadData(url);

This just basically downloads the html and into a byte array. This is what I want. The problem however is that the links within the html are most of the time relative, not absolute.

I basically want to append whatever the full http://domain.is before the relative link as to convert it to an absolute link that will redirect to the original content. I'm basically just concerned with href= and src=. Is there a regex expression that will cover some of the basic cases?

Edit [My Attempt}:

 public static string RelativeToAbsoluteURLS(string text, string absoluteUrl)
    {
        if (String.IsNullOrEmpty(text))
        {
            return text;
        }

        String value = Regex.Replace(text, "<(.*?)(src|href)=\"(?!http)(.*?)\"(.*?)>", "<$1$2=\"" + absoluteUrl + "$3\"$4>", RegexOptions.IgnoreCase | RegexOptions.Multiline);

        return value.Replace(absoluteUrl + "/", absoluteUrl);
    }

Answer 1

I think url is of type string. Use Uri instead with a base uri pointing to your domain:

Uri baseUri = new Uri("http://domain.is");
Uri myUri = new Uri(baseUri, url);

System.Net.WebClient client = new System.Net.WebClient();
byte[] dl = client.DownloadData(myUri);

Answer 2

While this may not be the most robust of solutions it should get the job done.

var host = "http://domain.is";
var someHtml = @"
<a href=""/some/relative"">Relative</a>
<img src=""/some/relative"" />
<a href=""http://domain.is/some/absolute""&gt;Absolute&lt;/a&gt;
<img src=""http://domain.is/some/absolute"" />
";


someHtml = someHtml.Replace("src=\"" + host,"src=\"");
someHtml = someHtml.Replace("href=\"" + host,"src=\"");
someHtml = someHtml.Replace("src=\"","src=\"" + host);
someHtml = someHtml.Replace("href=\"","src=\"" + host);

Answer 3

You should use HtmlAgility pack to load the HTML, access all the hrefs using it, and then use the Uri class to convert from relative to absolute as necessary.

See for example http://blog.abodit.com/2010/03/a-simple-web-crawler-in-c-using-htmlagilitypack/

Answer 4

You could use the HTMLAgilityPack accomplish this. You would do something along these (not tested) lines:

• Load the url
• Select all links
• Load the link into a Uri and test whether it is relative If it relative convert it to absolute
• Update the links value with the new uri
• save the file

Here are a few examples:

http://stackoverflow.com/questions/2719353/relative-to-absolute-paths-in-html-asp-net

http://htmlagilitypack.codeplex.com/wikipage?title=Examples&referringTitle=Home

http://blog.abodit.com/2010/03/a-simple-web-crawler-in-c-using-htmlagilitypack/

Answer 5

The most robust solution would be to use the HTMLAgilityPack as others have suggested. However a reasonable solution using regular expressions is possible using the Replace overload that takes a MatchEvaluator delegate, as follows:

var baseUri = new Uri("http://test.com");
var pattern = @"(?<name>src|href)=""(?<value>/[^""]*)""";
var matchEvaluator = new MatchEvaluator(
    match =>
    {
        var value = match.Groups["value"].Value;
        Uri uri;

        if (Uri.TryCreate(baseUri, value, out uri))
        {
            var name = match.Groups["name"].Value;
            return string.Format("{0}=\"{1}\"", name, uri.AbsoluteUri);
        }

        return null;
    });
var adjustedHtml = Regex.Replace(originalHtml, pattern, matchEvaluator);

The above sample searches for attributes named src and href that contain double quoted values starting with a forward slash. For each match, the static Uri.TryCreate method is used to determine if the value is a valid relative uri.

Note that this solution doesn't handle single quoted attribute values and certainly doesn't work on poorly formed HTML with unquoted values.

Answer 6

Uri WebsiteImAt = new Uri(
       "http://www.w3schools.com/media/media_mimeref.asp?q=1&s=2,2#a");
string href = new Uri(WebsiteImAt, "/something/somethingelse/filename.asp")
       .AbsoluteUri;
string href2 = new Uri(WebsiteImAt, "something.asp").AbsoluteUri;
string href3 = new Uri(WebsiteImAt, "something").AbsoluteUri;

which with your Regex-based approach is probably (untested) mappable to:

        String value = Regex.Replace(text, "<(.*?)(src|href)=\"(?!http)(.*?)\"(.*?)>", match => 
            "<" + match.Groups[1].Value + match.Groups[2].Value + "=\""
                + new Uri(WebsiteImAt, match.Groups[3].Value).AbsoluteUri + "\""
                + match.Groups[4].Value + ">",RegexOptions.IgnoreCase | RegexOptions.Multiline);

I should also advise not to use Regex here, but to apply the Uri trick to some code using a DOM, perhaps XmlDocument (if xhtml) or the HTML Agility Pack (otherwise), looking at all //@src or //@href attributes.