Database for Large number of 1kB data chunks (MySQL?)

Question

I have a very large dataset, each item in the dataset being roughly 1kB in size. The data needs to be queried rapidly by many applications distributed over a network. The dataset has more than a million items (so 500 million+ 1kB data chunks).

What would be the best method to storing this dataset (need to allow adding more items, and reading them rapidly, but never modifying already added data)? Would using a MySQL DB using the binary blob format be appropriate?

Or should each of these be stored as files on a file system?

edit: the number is 1 million items now, but needs to be able to scale to well over 500 million items easily.

Answer 1

That's one GB of data. What are you going to use the database for?

That's definitely just a file, read it into ram when starting up.

Scaling to 500Million is easy. That just takes some more machines. Depending on the precise application characteristics, you might be able to normalize or compress the data in ram.

You might be able to keep things on disk, and use a database, but that seriously limits your scalability in terms of simultaneous access. You get 50 disk accesses/sec from a disk, so just count how many disk you need.

Answer 2

If you need to retrive saved data then storing in files is certainly not a good idea.

MySQL is a good choice. But make sure you have right indexes set.

Regarding binary-blob. It depends on what you plan to store. Give us more details.

Answer 3

Since there is no need to index anything inside the object. I would have to say a filesystem is probably your best bet not a relational database. Since there's only an unique ID and a blob, there really isn't any structure here, so there's no value to putting it in a database.

You could use a web server to provide access to the repository. And then a caching solution like nginx w/memcache to keep it all in memory and scale out using load balancing.

And if you run into further performance issues, you can remove the filesystem and roll your own like Facebook did with their photos system. This can reduce the unnecessary IO operations for pulling unneeded meta-data from the file system like security information.