Does anyone know where there is a recipe for serializing data and preserving its order in the output?

Question

I am working with a set of data that I have converted to a list of dictionaries

For example one item in my list is

{'reportDate': u'R20070501', 'idnum': u'1078099', 'columnLabel': u'2005',
 'actionDate': u'C20070627', 'data': u'76,000', 'rowLabel': u'Sales of Bananas'}

Per request

The second item in my list could be:

 {'reportDate': u'R20070501', 'idnum': u'1078099', 'columnLabel': u'2006',
 'actionDate': u'C20070627', 'data': u'86,000', 'rowLabel': u'Sales of Bananas'}

The third item could be:

 {'reportDate': u'R20070501', 'idnum': u'1078100', 'columnLabel': u'Full Year 2005',
 'actionDate': u'C20070627', 'data': u'116,000', 'rowLabel': u'Sales of Cherries'}

The fourth item could be:

 {'reportDate': u'R20070501', 'idnum': u'1078100', 'columnLabel': u'Full Year 2006',
 'actionDate': u'C20070627', 'data': u'76,000', 'rowLabel': u'Sales of Sales of Cherries'}

The reason I need to pcikle this is because I need to find out all of the ways the columns were labeled befoe I consolidate the results and put them into a database. The first and second items will be one row in the results, the third and fourth would be the next line in the results (after someone decides what the uniform column header label should be)

I tested pickle and was able to save and retrieve my data. However, I need to be able to preserve the order in the output. One idea I have is to add another key that would be a counter so I could retrieve my data and then sort by the counter. Is there a better way?

I don't want to put this into a database because it is not permanent.

I marked an answer down below. It is not what I am getting, so I need to figure out if the problem is somewhere else in my code.

Answer 1

Python does not retain order in dictionaries.

Maybe a better choice would be to use a list of tuples?

[('reportDate', u'R20080501'), ('idnum', u'1078099'), ...etc]

You can use the built in dict() if you need to convert this to a dictionary later.

Answer 2

The Python dict is an unordered container. If you need to preserve the order of the entries, you should consider using a list of 2-tuples.

Another option would be to keep an extra, ordered list of the keys. This way you can benefit from the quick, keyed access offered by the dictionary, while still being able to iterate through its values in an ordered fashion:

data = {'reportDate': u'R20070501', 'idnum': u'1078099', 
        'columnLabel': u'2005', 'actionDate': u'C20070627', 
        'data': u'76,000', 'rowLabel': u'Sales of Bananas'}
dataOrder = ['reportDate', 'idnum', 'columnLabel', 
             'actionDate', 'data', 'rowLabel']

for key in dataOrder:
    print key, data[key]

Answer 3

So what's wrong with pickle? If you structure your data as a list of dicts, then everything should work as you want it to (if I understand your problem).

>>> import pickle
>>> d1 = {1:'one', 2:'two', 3:'three'}
>>> d2 = {1:'eleven', 2:'twelve', 3:'thirteen'}
>>> d3 = {1:'twenty-one', 2:'twenty-two', 3:'twenty-three'}
>>> data = [d1, d2, d3]
>>> out = open('data.pickle', 'wb')
>>> pickle.dump(data, out)
>>> out.close()
>>> input = open('data.pickle')    
>>> data2 = pickle.load(input)
>>> data == data2
True