Transforming deeply nested dictionary to 1D dictionary with Python

Question

I have some deeply randomly nested dictionary as follows.

{'CompilationStatistics': {'CodeGeneration': {'EndTime': '2010-04-21T14:03:11',
                                              'StartTime': '2010-04-21T14:03:11',
                                              'StepList': {'EliminatingDuplicates': {'EndTime': '2010-04-21T14:03:11',
                                                                                     'NumberOfFilesEliminated': '14',
                                                                                     'StartTime': '2010-04-21T14:03:11'},
                                                           'ModuleGenerator': {'EndTime': '2010-04-21T14:03:11',
                                                                               'StartTime': '2010-04-21T14:03:11'},
                                                           'Munger': {'EndTime': '2010-04-21T14:03:11',
...

How can I transform this into 1D dictionary as follows.

dict["CompilationStatistics_CodeGeneration_EndTime"] = '2010-04-21T14:03:11'
dict["CompilationStatistics_CodeGeneration_StartTime"] = '2010-04-21T14:03:11'
...
dict["CompilationStatistics_directory] = "/abc"

Answer 1

Hmmm, interesting...

This is a perfect opportunity for a recursive function. Here's one that, according to my preliminary tests, will do the job:

def convert(dct_in, dct_out=None, prefix='', sep='_'):
    if dct_out is None:
        dct_out = {}
    if prefix:
        prefix += sep
    for k, v in dct_in.iteritems():
        k_str = prefix + k
        if isinstance(v, dict):
            convert(v, dct_out, k_str, sep)
        else:
            dct_out[k_str] = v
    return dct_out

Answer 2

Recursively:

def flatten_dict(d):
    subdicts = (([(k+"_"+k2, v2) for k2,v2 in flatten_dict(v).iteritems()]
                 if isinstance(v, dict)
                 else [(k,v)])
                for k,v in d.iteritems())
    return dict((k,v) for sd in subdicts for k,v in sd)

Answer 3

Simplest is to do it recursively:

import collections

def flattendict(d, prefix=()):
  r = {}
  for k, v in d.iteritems():
    pk = prefix + (k,)
    if isinstance(v, collections.Mapping):
      r.update(flattendict(v, pk))
    else:
      r['_'.join(pk)] = v
  return r

Here's an example use:

d = {'foo': 'bar',
     'baz': {'fie': 'foo', 'zip': 'zap'},
     'bam': {'fie': 'foo', 'zip': {'zap': 'zup', 'mep': 'mop'}},
    }
print flattendict(d)

producing (possibly, of course, in different order)

{'baz_zip': 'zap', 'bam_fie': 'foo', 'foo': 'bar', 'bam_zip_mep': 'mop',
 'baz_fie': 'foo', 'bam_zip_zap': 'zup'}