How to fetch Android Market data when there is no API ?

Question

As far as I can tell, there is no API (official or unofficial) to access information about Apps on the Android Market (info such as Title, Icon, Description, Downloads, Comments, etc..) However, there are a few websites that have managed to compile this information anyway.

My question is: How is this being done?

A couple of guesses.

• Manually (not likely, due to volume)
• Implemented the protocol used by the official app (by decompiling or observing packets)
• Running some kind of debugger with the official app to extract the live data
• Using some kind of automation + screen shots + text recognition (seems complicated)
• Modifying the (open) source to write a log file

I'm just curious how this was done because I sure can't figure it out. Thanks.

Answer 1

As far as I know, they are using Protocol Buffers. You'll have to sniff phone traffic while it's accessing the market connected over wifi.

Answer 2

Another option to capture data is using the tcpdump tool that comes with some custom images (like Cyanogenmod and others). You can capture all the packets and then analyze them using Wireshark. This route is easier than hijacking wifi.

Checkout this splendid posts to clear out some things:

• Gathering packets with tcpdump from Android
• Downloading Market applications without the vending app

Update I have been working on this a bit more and @gsmd is right and they use Protocol Buffers to transfer the information and with the technique described above you can capture the packets. Then you'll need to dig into protobuffers to understand the messages.

Answer 3

Tried wireshark after getting the tcpdump, but couldn't figure out how to to make sense of it. Marc do you have any update on deciphering the tcp dump. Were you able to construct the protocol buffer message of the packets?

Trying to find if newer versions of app is available, via bypassing the market but literally stuck trying to get updated info about app without going through the market UI Cryket and Androlib seem to be up to date with the latest version info about all apps.

I think there must be another way to get this information without going the tcp dump/wireshark . TrackDog app is able to find this information on unrooted phones. Perhaps they are doing what alex suggested. Did anybody have any luck with sniffing market traffic over wifi.

Answer 4

There is a Java project on Google code which attempts to build an open source api for the market: http://code.google.com/p/android-market-api/

There're also some JRuby bindings for it: http://github.com/jberkel/supermarket.

However they both require a Google/Market account, and the API is only partially implemented (searching, comments and images at the moment). Also, the reconstructed protocol buffer descriptor (market.proto) could be used to generate bindings for languages other than Java.