Gigantic arrays

views:

101

answers:

+1 Q:

I'm planning to store gigantic arrays into serialized files and read from them in order to display data within them. The idea is to have a simple, document-oriented, filesystem database. Can anyone tell me if this would be a performance issue? Is it going to be slow or very fast?

Is it worth, filesystem is always really faster?

+3 A:

It will be very slow. Serializing and unserializing always requires reading and processing the whole array, even if you need only a small part.

Thus you are better of with using a database (like MySQL). Or if you need to only access key/value use APC/memcached.

nikic 2010-10-27 16:40:05

But that's not how a filesystem database works... It uses directories and files to organize the data instead of an array or other complex data structure. So there's no serialization/deserialization needed (well, above what you would need for MySQL/etc anyway)... I'm not arguing that it's right to implement, but for some use cases (LARGE amounts of sparse data, needing access from shell scripts, etc) it **may** make sense (now, these are few and far between). But it's by no means "very slow"...

ircmaxell 2010-10-27 16:47:49

@ircmaxell: Well the OP talks about serializing large arrays to the filesystem. That certainly doesn't make sense. He doesn't say anything about directories and files.

nikic 2010-10-27 16:50:57

@nikic: whoops. I must have skipped right over that... I still think the comment is valid (aside from that one line) otherwise. But I agree, there's no reason to store complex data structures in a serialized file yourself unless you're lazy...

ircmaxell 2010-10-27 16:53:21

+3 A:

You'll be much better off using a "proper" database - it's what they're designed for. If your data is really in a document-oriented format, consider CouchDB.

Craig A Rodway 2010-10-27 16:41:59

Unfortunately, i don't want to install anything. I need something that runs just with PHP. Simple and fast.

Jamie 2010-10-27 16:48:37

The PHP-only solution may be simple. But fast entirely depends on your dataset, operating system, filesystem, and disk architecture.

Craig A Rodway 2010-10-27 16:50:56

I think you could implement this without many performance issues, just so long as your arrays don't take forever to (un)serialize and you are able to lookup your files efficiently. How do you plan on looking up which file to read, btw?

Is it worth, filesystem is always really faster?

No, this method is not always faster, in fact you'd probably get better performance using some sort of db or cache with what you're trying to do.

PMV 2010-10-27 18:01:34

Is serialization too slow? What about json_encode?

Jamie 2010-10-27 19:34:05

I'm using file_put_contents and file_get_contents for writing/reading.

Jamie 2010-10-27 19:34:29

How big are your arrays?

PMV 2010-10-27 19:52:49

Quite big, a multidimensional array with +- 5,000 entries.

Jamie 2010-10-27 22:20:18

ansaurus

tags:

views:

answers:

Gigantic arrays

related questions