Hello I want to screen scrape a site like yelp to get phone numbers of italian restaurants.. I created a simple program to do just what I wanted but they blocked my servers ip
I am using php to do it. How can I get past the ip block?
I've heard about programs like screen-scraper, but I still haven't used it yet
What is the best way to do it? and is it possible to use screen-scraper with php?
Please note* this is for a personal project I'm working on, its not to create a business out of it
If you're doing this for commercial gain, stop right where you are. See if you can find licensed means to get at the same data, or pound the pavement yourself. Some companies intentionally inject mistakes or identifiable typos into their information as a way to catch people like you and will take legal steps to protect their intellectual property (even though that info is completely free if collected any other way). Being cheap can sometimes end up being very expensive.
If you're not doing this for commercial gain (and you just really love Italian food), move servers or wait until the IP block lifts (which may be never). Rewrite your code and put a massive rate-limiter on your requests (emulate a user and get one page every 5-10 seconds or so. Scrape the site over several days in short sessions. If they see too many requests from a single IP over too short a time, they will blacklist you again. If you were them, you would too.
If you only want phone numbers, there's probably an easier way to get that info, all on one page. Try a Yellow Pages sort of site. Look up Italian restaurants in your area. Save the whole page. You then have the numbers.
There may be another site that has this info available via an API, too - that way you don't have to break any terms of service. Poorly written or aggressive scraping scripts can temporarily damage webservers - there IS a reason sites block these actions.