views:

164

answers:

2

I'm using a NSKeyedArchiver to encode a big object graph (76295 objects.) It takes a lot of time but even worse the NSKeyedArchiver doesn't give back all its memory.

After using the Leak check, the code doesn't appear to leak at all but for some reason the encoding doesn't give back all memory after it's done.

Calling the encode method several times does make it worse, more and more memory is being eaten away.

You got any suggestions I could like at?

P.S. A database (sqlite) or CoreData aren't alternatives because they seem to scale very poorly with an big object graph like the one mentioned above.

I would prefer a solution using NSKeyedArchiver

+1  A: 

NSKeyedArchiver doesn't actually encode objects. It simply walks the graph and calls the encode methods of each instances in the graph. Therefore, the most likely source of a leak is inside a custom coder method that you wrote to archive one of your custom classes. You might want to do test archives of individual classes to see if one of them leaks.

The problem with using NSKeyedArchiver to archive a large complex graph is that the entire graph is piled into memory at once. One leak in one classes coder can explode if a lot of instances of that class are archived. If you have 76,000+ objects and just a couple of thousand leak a few bytes each, that will add up in a hurry.

I must add that I have never encountered or even read of a situation in which Core Data did not perform better than an archive for complex graphs regardless of size. Core Data was created specifically to handle just this sort of problem.

If you've tried Core Data and it bogged down, it is probably because you used to much entity inheritance. Since Core Data uses single table inheritance on the store side in SQL, all the descendants of an entity end up in the same table and that bogs things down. Remember, entities are separate from the classes they model so you can have class inheritance without having entity inheritance. That gives you the coding advantages of inheritance without the speed penalty of entity inheritance.

TechZen
I've done some testing, directly using sqlite to store the 111000 objects takes up to 3 minutes. Way too slow; adding an additional layer on top of that (CoreData) will only slow it down even more, sadly.
Ger Teunis
How often do you actually create and store 111,000 new objects? You might do so once when the launches the first time but most of the time, you would be just reading from a store of those objects. Core Data doesn't have to load in all 111,000 objects, just those immediately needed. Neither does it have to have all objects in memory in order to save one. Core Data is fast. I've replaced simple combinations of dictionaries and arrays with Core Data and seen an obvious speed increase where I expected none. Really cannot think of case in my experience in which Core Data did worse on a large graph.
TechZen
Sadly all the time. No real updates just complete new sets.
Ger Teunis
A: 

Seems the memory is given back to the system slowly at a later interval. So no real memory leak is there.

For anyone else: try look at CoreData or sqlite3 directly. If you use sqlite3 directly make sure you encapsulate the complete set of queries in an transaction; it will dramatically increase data throughput.

More info regarding sqlite speed optimization: http://web.utk.edu/~jplyon/sqlite/SQLite_optimization_FAQ.html

Ger Teunis