Will providing APIs help deter screen scraping?

Question

I have been thinking quite a bit here lately about screen scraping and what a task it can be. So I pose the following question.

Would you as a site developer expose simple APIs to prevent users from screen scraping, such as JSON results?

These results could then implement caching, and they are much smaller for traffic than the huge amounts of markup that could potentially be downloaded.

I am not looking at prevention, but deterring scraping.

---

Scraping Bandwidth Sample
((users * (% / 100)) * ((freq * 60) * 24)) * filesize

• users: 200,000
• % of users using utility: 5
• filesize: 1kb
• freq: 1 minute

Formula:

((users * (% / 100)) * ((freq * 60) * 24)) * filesize

10,000 * 1440 * 1

14400000kb or 13.73291015625gb

Assuming your JSON result is 200 bytes that's now (10,000 * 1440 * 0.2) or 2.74658203125gb a day.

That's a change of about 11gb of traffic a day.

---

My Stack Overflow profile is 96k for reference.

---

The reason for this question prompted asking for a JSON result from users profiles:
http://stackoverflow.uservoice.com/pages/general/suggestions/101342-add-json-for-user-information

I wanted to find out if other developers would expose this type of API, and if it is worth your time to provide these APIs to reduce bandwidth.

Answer 1

If you want to provide an open model that people can develop solutions ontop of your site, then yes you should provide an API. Screen scrapping is a method of hostile integration, and should only be used in last resort.

Answer 2

Screen scraping is not realistically preventable. Providing an API, while nice to those who consume your data, can't prevent it. Since the data ultimately has to be human readable, it therefore is machine readable. You would be better off spending your energy working on your site and not working for those who would consume your data (legally or not).

wget, perl, regular expressions is the common mechanism for scraping data.

Answer 3

Providing an API should definitely reduce the amount of screen scraping that gets done against your site. Using a good REST API is much easier and safer than screen scraping. Screens can change without notice, and that makes screen scraping code much harder to maintain. As a developer, if I need information from a site, I'd never scrape the site if the same information was available through an API.

Answer 4

If you want to encourage people to integrate with your site or it is popular enough for this to be a problem (so that you are forced to allow people to integrate with it), then by all means provide an API. If your API is adequate and easy to use, then people will prefer it to screen scraping. If your API is inadequate or harder to use than a screen scraper then you may still have the problem.

Answer 5

If it's easier for technical users to use an API than it is for them to screen-scrape, they will do so. Even better so, if you can encourage people to use your APIs instead of screen-scraping, you should have a much easier time monitoring traffic, because the automated user-agents are clearly distinguished from the browser user-agents.

A RESTful JSON interface is a good choice, because it can be scripted from any other language fairly easy (show me a language that doesn't have a JSON parser and I'll show you a language nobody cares about).

Answer 6

Most developers will choose the technology to use for their own reasons. So if you provide an API that is easier than what they use to scrape your screens, then some unkown percent will move to it. Bandwidth reduction will probably be very low on their lists of considerations.

Since you haven't specified what's being scraped for, we can't help you guess what kind of API to provide, or what proportion will use it.

One very common tool for scraping that's hard to deflect is using Excel or some other product that makes the scrape painless.

If your intent is just to minimize the pain (which one might infer from your question), then by far the most useful thing to do is to query they scrapers - more useful than querying SO, anyway.

You might check woot.com and see what they provide on an RSS feed to unburden the web http server.