Django initial_data.json memory issues

Question

I have a huge (400MB) initial_data.json file. Django's eating up all my memory trying to load this file.

Suggestions? Surely someone out there has large initial_data files they use. I'd consider converting this into an xml file, but I don't know if Django will try to load all that into memory as well, and I'm not yet ready to try it out without being sure.

Answer 1

When you load a json file, django routes it through simplejson, which eventually does this:

    return loads(fp.read(),  ...) 

(From django.utils.simplejson)

In other words, the entire file is read (fp.read()) before it's passed on to the decoder. It seems possible (but not trivial) to modify simplejson to read file a bit at a time.

However, the XML serializer uses pulldom, which apparently "streams" the XML file. As such, it should be much more memory efficient.