Best way to deploy an application that comes with lots of data

Question

I'm working on an Android application that will come with lots of data. I'd have expected to just distribute the SQLite database, but it's clear from researching this that the database can be included, but must be copied from raw to data, which if there's a good few MB of data will take up space needlessly. For my app the data is all text/numeric/date, and in a relational database would normally take up four tables. So pretty simple, but having two copies of it seems v wasteful.

The options I see are:

1. Tolerate the data being duplicated (it pains me)
2. On installation the app downloads the data over HTTP (I'd then have a potential challenge with the load), probably Google App Engine.

Any other solutions worth considering?

Answer 1

Not a proper answer, but on the subject of option 1: remember an apk is a zip, so a text file, containing the same words over and over, would become a lot smaller.

Answer 2

If you package your data into your res/raw folder, it will indeed be duplicated and unfortunatley cannot be removed once the phone is done with it ie after first run.

You could experiment which is a smaller in size - packaging the data as a csv or xml file or as a precompiled DB in the res/raw folder and if acceptible go with the best. If you went with the csv or xml file option, you could run a parser on first run and process the data into a DB.

Something I only found recently but the res/raw folder can only handle files of around 1mb, this is one solution I came across for packaging a large DB and moving it on first run, it doesn't get over the repition of data within the app:

Database Populating Solution

Here is an alternative from the android blogspot, the third one down maybe exactly what you are looking for:

The downloader activity runs at the beginning of your application and makes sure that a set of files have been downloaded from a web server to the phone's SD card. Downloader is useful for applications that need more local data than can fit into an .apk file.

Downloader Blogspot

Possibly adapt this example - download the DB file from the server and then move it into your database folder.

Answer 3

I was confronted to the same kind of situation: I am writing an app which has to access - locally – a database containing mainly text and weighing more than 20MB (7MB zip compressed).

The best solution in my opinion for this kind of problem is to download the zipped compiled database on first execution of the app. The advantages of this solution:

• The APK stays light. This is a good because:
  • very large APKs may discourage some potential users
  • testing is faster, as uploading a 7MB+ APK to an AVD (as I did initially) is pretty SLOW.

• There is no duplicate of your data (if you take care of deleting the ZIP archive after extracting it

• You don’t fill up your users phones internal memory. Indeed, your app should download the database directly to the SD CARD. Including big files in res/raw folders would cause much trouble to all users who run Android < 2.2, as they won’t be able to move the app to the SD CARD (I’ve seen several negative user comments due to this on some apps that use a few MBs on the internal memory).

One last thing: I don’t recommend including a CSV or XML and populate the database from it on first run. The reason being this would be very SLOW, and this kind of processing is not supposed to be done by every client.