python simpleJSONDecoder and complex JSON issue

Question

In a unit test case that I am running, I get a KeyError exception on the 4th json object in the json text below because the piece of code responsible for decoding is looking for an object that isn't there, but should be.

I went through the sub-objects and found that it was the "cpuid" object that causes the problem. When I remove it and run the test, it works fine.

def _make_report_entry(record):

    response = self.app.post(
        '/machinestats',
        params=dict(record=self.json_encode([
                    {"type": "crash", "instance_id": "xxx",
                     "version": "0.2.0", "build_id": "unknown",
                     "crash_text": "Gah!"},
                    {"type": "machine_info", "machine_info": "I'm awesome.",
                     "version": "0.2.0", "build_id": "unknown",
                     "instance_id": "yyy"},
                    {"machine_info": "Soup", "crash_text": "boom!",
                     "version": "0.2.0", "build_id": "unknown",
                     "instance_id": "zzz", "type": "crash"},
                    {"build_id" : "unknown", "cpu_brand" : "intel",
                     "cpu_count" : 4,
                     "cpuid": {
                               "00000000":
                                    {"eax" :123,"ebx" :456,
                                     "ecx" :789,"edx" :321},
                               "00000001":
                                    {"eax" :123,"ebx" :456,
                                     "ecx" :789,"edx" :321}},
                     "driver_installed" : True,
                     "instance_id" : "yyy",
                     "version" : "0.2.0",
                     "machine_info" : "I'm awesome.",
                     "os_version" : "linux",
                     "physical_memory_mib" : 1024,
                     "product_loaded" : True,
                     "type" : "machine_info",
                     "virtualization_advertised" : True}
                    ])))

In the piece of code being tested, I use simplejson.JSONDecoder from django.utils to decode the JSON. When I log the decoded output for the above JSON that gets passed to my decoding function, I get this:

root: INFO: {u'instance_id': u'xxx', u'type': u'crash', u'crash_text': u'Gah!', u'version': u'0.2.0', u'build_id': u'unknown'}

root: INFO: {u'build_id': u'unknown', u'instance_id': u'yyy', u'version': u'0.2.0', u'machine_info': u"I'm awesome.", u'type': u'machine_info'}

root: INFO: {u'build_id': u'unknown', u'machine_info': u'Soup', u'version': u'0.2.0', u'instance_id': u'zzz', u'crash_text': u'boom!', u'type': u'crash'}

root: INFO: {u'eax': 123, u'edx': 321, u'ebx': 456, u'ecx': 789}

On the last JSON object, only the object within the JSON cpuid object is being passed to my decoding function. Because my decoding function is expecting the the other objects (e.g., 'type', 'instance_id', etc.), I get a KeyError exception.

[Sorry for the earlier unnecessarily long post, I hope that this will narrow it down a bit more]

Answer 1

Copying and pasting what you're passing to self.json_encode, and using it as an argument of json.dumps (after an import json in Python 2.6), works just fine. So it seems the bug may be in the json_encode method you're not showing us: what else does it do, besides just calling json.dumps...? (or simplejson.dumps if you're using a Python < 2.6, of course).

Edit: using json_encode = json.JSONEncoder().encode (as the OP just posted, except that's using the older simplejson as I had mentioned as a possibility) also works fine. The incomplete stack-trace also posted as part of the Q's large edit suggests that the error comes in the decoding part, perhaps through misuse of some model (can't tell, as we don't see the models) -- as the OP mentioned he's now posted a lot more info, and yet it's still not enough to debug the problem.

This strongly suggests that it would be worthwhile for the OP to try and simplify the problem a little at a time until the last incremental simplification makes the bug disappear -- that usually strongly hints at what the bug may be, but even if it doesn't, posting the tiniest way to reproduce the bug plus the info that the bug will disappear if a miniscule epsilon of code is further removes, may help "third party observers" like us all assist in the debugging. SO is not really a platform designed for collective debugging (works better for questions and answers, what it was designed for) but I don't think it breaks SO's rules to try and use it for this different purpose.

Answer 2

Last 2 lines in traceback:

File "...j_report/src/jreport/machinestats.py", line 77, in _make_report_entry
entry_type=record['type']

You have now TWO versions of def _make_report_entry(record):

Note that the first few lines of the traceback are muttering about decode, not encode.

What has the first/original version to do with the problem?

You now say "Because my decoding function is expecting the the other objects (e.g., 'type', 'instance_id', etc.), I get a KeyError exception."

So perhaps your decoding function is being called recursively and is expected by the caller to be able to handle ANY structure, not just ones with 'type' etc.