Crawling itunes.apple.com

Question

Hi I am trying to crawl the apple itunes website. I am getting output in binary format. For example

curl -A "mozilla/5.0" http://itunes.apple.com/us/app/the-far-islands-by-john-buchan/id327765949?mt=8

returns binary.

Can anybody please tell me what i am missing?

Thanks

Answer 1

You're getting binary back because the page you cited isn't returning HTML/XML, it's returning an Apple WebObject. From wget:

wget http://itunes.apple.com/us/app/the-far-islands-by-john-buchan/id327765949?mt=8
--2010-08-03 12:38:14--  http://itunes.apple.com/us/app/the-far-islands-by-john-buchan/id327765949?mt=8
Resolving itunes.apple.com... 17.250.237.16
Connecting to itunes.apple.com|17.250.237.16|:80... connected.
HTTP request sent, awaiting response... 200 Apple WebObjects
Length: 22900 (22K) [text/html]
Saving to: `id327765949?mt=8'

100%[======================================>] 22,900      --.-K/s   in 0.05s   

2010-08-03 12:38:14 (440 KB/s) - `id327765949?mt=8' saved [22900/22900]

See the good old Wikipedia for more info, but if you want to crawl it, you may need to use something that simulates a browser and thus can interpret it - maybe watir would work.