Handling splitting and joining of objects in key-value storage?

Question

Hi everybody,

After getting some help on how to measure fysical/actual size of memcached objects to prevent them from growing too large, I have thought about the next step - implementing a sharding/splitting function that transparently splits large objects into smaller pieces upon storage and glues them together as one big object when requesting them. Basically it should do everything behind the scenes automatically that needs to be done to keep memcached happy.

What's an appropriate way to handle splitting of array, objects or whatever kind of objects?

I am using PHP in my webapp, but for this case, I would be quite happy with a general approach with some psuedo-code to point me in the right direction.

Thanks a lot!

Answer 1

In the other question, serialize is used to measure the stored length of the object. If you're hitting the default one meg limit on object size, and you need to split things up, you can simply use serialize, split the resulting string into appropriate chunks, and then store the chunks. Later you can join them back together again and unserialize the result.

That being said... seriously, if your object, serialized, is a meg in size, you might want to reconsider how you're storing things. PHP's serialize can be a bit slow (compared to, say, json_encode), and throwing a meg or more of data at it is not likely to be the best or fastest way to do whatever it is you're doing.

If you're implementing memcached and sharding as a performance mechanism, I urge you to stop right now unless you've already used a tool like Xdebug to profile your code and have eliminated all other bottlenecks.