heavy iTunes Connect scraping

Question

I'm looking at different options to get the sales reports and other data out of the iTunes Connect website. Since Apple doesn't provide an API, all the solutions I found are based on scraping the page.

As I need the information for a product that we offer, I'm not that happy to give all the iTunes accounts to a 3rd party service. This is why I want to scrape it myself or use a product that runs on our servers.

My questions are:

• does someone have experience how frequent apple is changing the web front-end?
• has someone experience in maximum request from one server to the site? I'm afraid of being baned by apple.
• anything else I have to have in mind that will cause serious trouble?

---

Just if someone is interested in the tools I looked at, here is a list:

Services:

• http://www.appfigures.com (has API)
• http://www.itunesapis.com
• http://www.appannie.com/
• http://www.heartbeatapp.com

Products:

• http://www.appclix.com (has a enterprise licence that runs on your own server, includes API. Tends to me more a mobile analytics tool in general)
• http://www.ideaswarm.com/products/appviz/ (Mac enduser app)

Open Source Tools:

• http://code.google.com/p/appdailysales/
• http://search.cpan.org/~bfoz/p5-WWW-iTunesConnect-1.14/lib/WWW/iTunesConnect.pm
• http://www.rogueamoeba.com/utm/2009/05/04/itunesconnectarchiver/
• http://github.com/kasatani/iphone-stats
• http://bfoz.net/projects/itc/
• http://sourceforge.net/projects/itunesanalytics/

UPDATE:

I started using Kirby's python script (http://code.google.com/p/appdailysales/) and it works very well.

Answer 1

does someone have experience how frequent apple is changing the web front-end?

I can't speak for all of iTunes Connect, only downloading daily sales reports. My script was rock solid and didn't require a single change between November 2009 and September 2010. This changed in September 2010 when Apple rolled out the new web site. This broke the old script, and a new one had to be written. Since rolling out the new web site, I make changes every few days to handle the tweaks from Apple. I'm hoping the tweaks will end soon.

Take a look at the download page for appdailysales.py. The dates will give you a general idea of how often I make changes to the script.

http://code.google.com/p/appdailysales/downloads/list

Again, this is only for daily sales reports. I'm not sure how frequently others areas of iTC change.

has someone experience in maximum request from one server to the site? I'm afraid of being baned by apple.

I've not experienced this, but my server runs the script only once a day. I frequently hit the iTC when working on the script, but not enough to cause a load on Apple's servers.

anything else I have to have in mind that will cause serious trouble?

I don't know what might get you in trouble with Apple, but one thing that does cause a serious headache is changes to the web site. While the new version of the web site makes screen scraping the site easier, it did involve writing a new script. Apple does not give you a heads up that they are changing something. You find out after the fact when something in your screen scraper breaks.

If you depend on the data daily, then you have to drop everything and make the necessary fixes. And there is nothing stopping Apple from rolling out another new site sometime in the future.

Hope that helps.

-KIRBY

Answer 2

Another tool to look at is iMacros.

Answer 3

appdailysales is the best tool out there that I have found.

I have modified it so that the script automatically puts the ITC data into a MySQL database instead of just saving the txt files. And as Kirby pointed out, I too only have it run once a day and everything appears to be working. Nothing has been blocked by Apple so far.

As for the script breaking, the one good thing is that Apple keeps daily sales reports for 14 days (last I checked). This means that if the script breaks, one has several days to fix the script and still get the daily sales reports.

Good luck.

Kevin