How can I check a url is publicly accessible?

Question

I have a web application that you can use to import information from another site by giving it a url. It's been pointed out that you could use this feature to access a private site that is hosted on the same web server.
So...
How can I check that a given url is publicly accessible (whether on the same web server or somewhere different)?

FIX: I ended up doing this:

    protected static bool IsHostWithinSegment(string Host)
    {
        Ping pinger = new Ping();
        string data = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
        byte[] buffer = Encoding.ASCII.GetBytes(data);
        PingOptions options = new PingOptions();
        options.Ttl = 1;

        PingReply reply = pinger.Send(Host, 1000, buffer, options);

        return reply.Status == IPStatus.Success;
    }

    private static Uri BindStringToURI(string value)
    {
        Uri uri;
        if (Uri.TryCreate(value, UriKind.Absolute, out uri))
            return uri;

        // Try prepending default scheme
        value = string.Format("{0}://{1}", "http", value);
        if (Uri.TryCreate(value, UriKind.Absolute, out uri))
            return uri;

        return null;
    }

The only requirement of mine that it doesn't fulfil is that some installations of our product will exist alongside each other and you won't be able to import information across them - I suspect this will require using a proxy server to get an extrenal view of things but as it's not a requirement for my project I'll leave it for someone else.

-- I've just realised that this does entirely solve my problem since all the publicly accessible urls resolve to virtual or routable ips meaning they hop.

Answer 1

Check the URL address, and see if it matches your server address?

edit: or check against a range of addresses...

But all this does not answer the question: could the client access it?

Maybe some script in the browser to check that the url is accessible, and informing the server of the result. But the user could edit the page, or simulate the result...

Have the client read the url contents and send it back to the server, instead of having the server fetch it?

Answer 2

Only two things spring to mind.

1. Have a trusted external server verify the visibility of the address (like an HTTP Proxy)
2. Check the DNS record on the site -- if it resolves to something internal (127.0.0.1, 10.*, 192.168.*, etc) the reject it -- of course, this might not work depending on how your internal network is set up

Not knowing if this is on a 3rd-party hosting solution or inside your/your company's internal network makes it hard to say which solution would be best; good luck.

EDIT: On second thought, I've canceled the second suggestion as it would still leave you open to DNS rebinding. I'll leave this here for that purpose, but I don't think it's a good idea.

That said, if you have some ability to control the network makeup for this server, then it should probably live in its own world, dedicated, with nothing else on its private network.

Answer 3

You are asking the wrong question. You should be asking, how can I limit access to a given URL so that only people on a certain network can access it?

The fact is, that you cannot test in the way that you wanted, because you likely do not have access to other sites on the same web server, in order to run a script that attempts to retrieve a URL. It is better to deny all access except the access that you wish to allow.

Perhaps a firewall could do this for you, but if you want more finegrained control, so that some URLs are wide open, and others are restricted, then you probably either need help from the web server software or you need to code this into the application that serves the restricted URLs.

If you are worried that your web application might be used to transfer data that comes from other servers protected by the same firewall which protects you, then you should change the application to disallow any URLs where the domain name portion of the URL resolves to an IP address in the range which is protected by the firewall. You can get that address range information from the firewall administrator.

This is only really a concern on in-house systems because in 3rd party data centers there should not be any private servers that don't have their own protection. In other words, if it is at your company, they may expect their firewall to protect the whole data center and that is reasonable, if a bit risky. But when you rent hosting from a 3rd party with a data center on the Internet, you have to assume that everything inside that data center is equally as potentially hostile as the stuff outside.

Answer 4

Run a traceroute (a series of pings with short TTL's to the address, if the firewall(s) is(are) one of the hops then it's visible from outside the organisation so should be acceptable.

System.Net.NetworkInformation has a ping class that should give you enough information for a tracert like routine.

This does sound like a big hole though, another approach should probably be considered. Preventing the machine that runs this prog. from accessing any other machine on the internal network may be better - a kind of internal firewall.

I've added a simple traceroute, since you like the concept:-

class Program
{
    static void Main(string[] args)
    {
        PingReply reply = null;
        PingOptions options = new PingOptions();
        options.DontFragment = true;
        Ping p = new Ping();
        for (int n = 1; n < 255 && (reply == null || reply.Status != IPStatus.Success); n++)
        {
            options.Ttl = n;
            reply = p.Send("www.yahoo.com", 1000, new byte[1], options);
            if (reply.Address != null)
                Console.WriteLine(n.ToString() + " : " + reply.Address.ToString());
            else
                Console.WriteLine(n.ToString() + " : <null>");
        }
        Console.WriteLine("Done.");
        System.Console.ReadKey();
    }
}

Should be good enough for a reliable local network.

Answer 5

Don't worry about the public accessibility of anyone else's web assets, that question does not have a definite answer in all cases. Just try not to compromise the access policy to your own (or your customer's etc.) web assets.

Use the existing access control mechanisms to control the web application's access. Don't just consult the access control mechanisms in order to duplicate them in the web application. That would be relying on the web application to refrain from using its full access - a false reliance if the web application ever gets compromised or if it simply has a bug in the access control duplication functionality. See http://en.wikipedia.org/wiki/Confused_deputy_problem.

Since the web application acts as a deputy of external visitors, treat it if you can as if it resided outside the internal network. Put it in the DMZ perhaps. Note that I'm not claiming that the solution is one of network configuration, I'm just saying that the solution should be at the same level at which it is solved if the visitor would try to access the page directly.

Make the web application jump through the same hoops the external visitor would have to jump. Let it fail to access resources the external visitors would have failed to access, too. Provide an error page that does not let the external visitor distinguish between "page not found" and "access denied".