How legal is screen scraping?

Question

I'm trying to build an iPhone application that gathers content from real estate websites to display it in a mashed-up and structured manner (mapping, price averages...etc)

I've stumbled upon many sites whose "Terms and Conditions" only allow downloading/re-using the data for personal purposes but not commercial ones. My intent is to have the iPhone application do all the work (when a user initiates a search query) of scraping the data and re-displaying it, of course with direct links to the original site page. In other words, I'm not crawling the website and storing it in my database and then re-aggregating its content to the user through some custom web service.

My question is: since a user of my application is initiating the search request and re-displaying the results from screen scraped sites (through my app) for their personal use, would this still be OK? or is this still considered commercial re-use of data?

Answer 1

If you have to ask, you must realize that you're already in a grey area.

I would consider what you are describing as commercial re-use, as you are effectively taking other people's hard work and making money off it. Every time they make an update to some content, you make a buck, even though they've explicitly said you can't use it in that way.

You will need to consult a lawyer for a definitive answer.

Answer 2

How you compare with Google site (or other search engine that matter)? I think if you service is a point of aggregation and link back to the original sites, you could use their public information. Maybe you can make an aggreement with his sites and gain some money for each click you redirect.

EDIT: I agree with @womp:

You will need to consult a lawyer for a definitive answer.

Answer 3

Most real estate data presented on realtors websites is actually aggregate data that they have licensed from a vendor. You might go down the route of trying to get it from the source.

You might start looking here to get info on what you are up against.

Answer 4

Probably not. By doing so you are essentially claiming the information is your and/or denying that site the opportunity to recoup expenses via advertising.

As for the legality, of course I am not a lawyer, but the legalities will vary depending on your jurisdiction, the jurisdiction of the site or sites in question and the terms and conditions of the site.

That all being said, what about the ethics? Would you want someone scraping your site in such a manner?

Lastly, how's this for a crazy idea: have you considered approaching those sites and asking if they would consider either allowing it or providing an API? If you somehow acknowledged them, linked to their site or so on they quite probably might. That way you avoid any possible angst.

Answer 5

Screen scraping itself is not illegal, to my knowledge. We've used it at my company with some of our intranet websites.

I'd google it and do some reading up on it. Just make sure you know the consequences before you make the leap.

Answer 6

Just remember eBay and all those aggregator projects who tried to filter out only negative feedbacks from thousands of them for a particular seller. Their terms prohibited reusage of their services in that manner. The last I checked there were no more such projects left. eBay lawyers got them all.

Answer 7

Unfortunately, if Google's experiences with newspapers and AP are any indication, you'll probably face issues, to put it mildly. As usual, TechDirt.com has relevant discussions:

http://www.techdirt.com/articles/20090504/0253274733.shtml

Answer 8

For intellectual properties, the property owner usually can override what's written in terms of services if you can get permission in writing. If you don't want to talk to the lawyer, or even if you do talk the lawyer, it might be worth a shot to just contact the real estate websites.

The realtors may be more interested in making sales (and for that purpose increasing visibility of the property) than protecting the rights to display listings -- as long as you make sure the agents get contacted.

Just explain them that you want to make money writing iPhone app and help them sell more houses.

Answer 9

The information is publicly available. You are not making commercial use of the information per se, you are providing a tool which can read that information.

As I see it, this is perfectly legitimate. Of course, as everyone else has pointed out, we are not lawyers.

Answer 10

A hot link is free to grab and link against.
So if you want:

<a href=http://jobs.com/software_devel_11.html&gt;Software Developer</a>

You can take the job title "Software Developer" and the link to the actual job.
However, if there are images and descriptions that are related to that link you may not take those as they are the publisher's content.
Google, Yahoo and the rest of the gang making their money by scraping the Internet.
So, if it was illegal, then they would have been charged as such.
However, those search bots, have to respect the robot.txt and if they don't, then there can be a lawsuit.

I have had the same problem while working on newjoblist.com

What I end up doing, was only grab the job titles and job links and not the descriptions.
Lawyers can't help you. Too complex and depends on each sites Policy.
Hope this helps.

Answer 11

ToS is just ToS.

it's not the law. you can face lawsuits, but highly rare in such situations where you are scraping. they will simply block your ip, or send a Cease and Desist letter, which you can ignore. then they will take you to court. These are extreme cases.

Scrape on my friend.

Answer 12

not they can't, I believe.

If they prohibit you, then they must sue all browser manufacturer as well; since what a browser do is basically download their html, process it, then view it to the user. Your program is just a custom browser that download their html, process it, then view it to the user.

Answer 13

I would imagine that screen scraping depends on what you are "saying" with the information. If I scrape a list of jobs at another site to plop on a map (which I do), but then I link to the site for more information and application information, then (A) I've just sent them business (absolutely no damage), and (B) all I've done in the case of each pin on the map is say "(Some site) has a job posting for (some position) with (some employer) that was posted on (some date), and you can get more information at (some URL)." WRITING those words would be completely legal, and placing the pins conveys no more information than that.

On the other hand, if you scrape a bunch of blogs and news sites to copy content (especially if it's verbatim) to display on your own site, maybe/never link/credit the other site, and rarely/never send traffic their way, then you're detracting from their potential profit from the information they produced, and it's obviously theft to some degree.

A good case in point came up at my job several years ago. We list news stories that pertain to our survey projects. One of the project managers wanted to make PDFs of relevant news stories and post them on our site. Obviously this was flat-out information theft, and I refused to do it. On the other hand, we've been posting titles, summaries and source information for years. We've have had no complaints, and this is likely because we force users to visit the site to read the whole story. All we've said is "Hey, so-and-so wrote an article titled such-and-such about this-or-that, and here it is."

In my opinion, and I'm sure this will usually be the case for the companies and courts involved, any case regarding screen scraping should be a matter of determining whether it was "profitable promotion" vs. outright theft. If the former were to be found illegal, it would mean that radio, television and other media (including Google) would be shut down entirely because they couldn't redistribute factual information for a profit.