Protecting my website content from being downloaded through cURL

Question

I heard about external access of a website using curl by the following code:

$curl_handle=curl_init();
curl_setopt($curl_handle,CURLOPT_URL,'http://example.com');
$buffer=curl_exec($curl_handle);
curl_close($curl_handle);

I want to protect my website from this external access. I am using PHP. How can I protect my web site?

Answer 1

This is no more dangerous than anyone accessing through your website with a browser; in fact, this is what curl is as far as your question is concerned: a web browser and nothing more.

If you have security vulnerabilities, they will be broken. If not, they won't. curl, however, does not amplify or diminish any vulnerabilities whatsoever; it is merely a way for a PHP script to access another website (typically through HTTP).

If you want to prevent content from being downloaded, you can block incoming HTTP requests, but this defeats the purpose of a web site, since web browsers make identical requests (and can also download content).

Answer 2

If you mean "how do I protect my website against being accessed by a program other than a browser", the answer is pretty much "you can't." A browser is just a program that sends HTTP requests. You can play the game of trying to reject HTTP requests that look like they don't come from a browser, but it's trivially easy for an arbitrary program (one using curl, or Perl/Python/Ruby libraries) to mimic the headers that a "real" browser sends.

Answer 3

You can't, if you want your users to be able to see it.

You could set up some system so that it won't serve up some content unless another related piece of content was very recently requested by the same user, but even that will not stop someone who really wants to get it...

Answer 4

I agree with waiwai, however, if you really want to 'protect' yourself against basic non-browser access, consider encrypting your content and decrypting it using, say, JavaScript when the site is accessed. Any browser that understands JavaScript (and has it enabled) will be able to display the content properly. But even encrypting the content can be worked around easily.

The setbacks are 1) it's not 'secure', 2) search engines may index your website incorrectly (afterall, they use cURL-like calls), 3) users who have JavaScript disabled are unable to access your content.

Overall, I'd say it's not worth it.