What is the difference between __reduce__ and __reduce_ex__ ?
I understand that these methods are for pickling/unpickling and have no relation to the reduce built-in function, but what's the difference between the 2 and why do we need both? ...
pickle
Can I pickle a python dictionary into a sqlite3 text field?
Any gotchas I should be aware of? Can I store it in a text field, or do I need to use a blob? (I'm not overly familiar with either pickle or sqlite, so I wanted to make sure I'm barking up the right tree with some of my high-level design ideas.) ...
module to create python object representation from xml
I'm searching for an easy to handle python native module to create python object representation from xml. I found several modules via google (one of them is XMLObject) but didn't want to try out all of them. What do you think is the best way to do such things? EDIT: I missed to mention that the XML I'd like to read is not generated b...
Why is marshal so much faster than pickle?
From this question and my own benchmarks it seems that the marshal module is about 20-30x faster than cPickle. Why is this so? What functionality does cPickle offer over marshal that justifies this? (Another way of putting it - why not always use marshal? Why do both of these modules exist?) ...
Serializing a Python object to/from a S60 phone
I'm looking for a way to serialize generic Python objects between a CherryPy-based server and a Python client running on a Symbian phone.. Since pyS60 doesn't implement the pickle module, how would you do it? I know about Cerealizer but it requires you to register classes before use (which I'd like to avoid) and doesn't look very mature...
Python human readable object serialization
Hi, i need to store Python structures made of lists / dictionaries, tuples into a human readable format. The idea is like using something similar to pickle, but pickle is not human-friendly. Other options that come to my mind are YAML (through PyYAML and JSON (through simplejson) serializers. Any other option that comes to your mind? T...
Lightweight pickle for basic types in python?
All I want to do is serialize and unserialize tuples of strings or ints. I looked at pickle.dumps() but the byte overhead is significant. Basically it looks like it takes up about 4x as much space as it needs to. Besides, all I need is basic types and have no need to serialize objects. marshal is a little better in terms of space but t...
ImportError: No module named copy_reg pickle
I'm trying to unpickle an object stored as a blob in a MySQL database. I've manually generated and stored the pickled object in the database, but when I try to unpickle the object, I get the following rather cryptic exception: ImportError: No module named copy_reg Any ideas as to why this happens? Method of Reproduction Note: Must do...
Why do I get unexpected behavior in Python isinstance after pickling?
Putting aside whether the use of isinstance is harmful, I have run into the following conundrum when trying to evaluate isinstance after serializing/deserializing an object via Pickle: from __future__ import with_statement import pickle # Simple class definition class myclass(object): def __init__(self, data): self.data = d...
How to recover a broken python "cPickle" dump?
I am using rss2email for converting a number of RSS feeds into mail for easier consumption. That is, I was using it because it broke in a horrible way today: On every run, it only gives me this backtrace: Traceback (most recent call last): File "/usr/share/rss2email/rss2email.py", line 740, in <module> elif action == "list": list(...
Bad Pickle get error
Hi, I have been using a flash card program called Mnemosyne which uses python script. A short time ago my database of flash cards became inaccessible after my computer froze and I had to shut it down manually. Whenever I try to load the data base containing my cards I get this error. Invalid file format Traceback(innermost last): Fi...
more efficient way to pickle a string
The pickle module seems to use string escape characters when pickling; this becomes inefficient e.g. on numpy arrays. Consider the following z = numpy.zeros(1000, numpy.uint8) len(z.dumps()) len(cPickle.dumps(z.dumps())) The lengths are 1133 characters and 4249 characters respectively. z.dumps() reveals something like "\x00\x00" (act...
EOFError in Python script
I have the following code fragment: def database(self): databasename="" host="" user="" password="" try: self.fp=file("detailing.dat","rb") except IOError: self.fp=file("detailing.dat","wb") pickle.dump([databasename,host,user,password],self.fp,-1) self.fp.close() selffp=f...
Using Python set type to implement ACL
Currently I have tables like: Pages, Groups, GroupPage, Users, UserGroup. With pickled sets I can implement the same thing with only 3 tables: Pages, Groups, Users. set seems a natural choice for implementing ACL, as group and permission related operations can be expressed very naturally with sets. If I store the allow/deny lists as pic...
How to hash a large object (dataset) in Python?
I would like to calculate a hash of a Python class containing a dataset for Machine Learning. The hash is meant to be used for caching, so I was thinking of md5 or sha1. The problem is that most of the data is stored in NumPy arrays; these do not provide a __hash__() member. Currently I do a pickle.dumps() for each member and calculate a...
Python - Save the context
Hi to all, I need to save the context of the program before exiting ... I've put all the needed stuff to an object that I've previously created a I tried many times to picke it, but no way !! I continuously have errors like : PicklingError: Can't pickle 'SRE_Match' object: <_sre.SRE_Match object at 0x2a969cd9c0> OSError: [Errno 1] Oper...
How do I upload pickled data to django FileField?
I would like to store large dataset generated in Python in a Django model. My idea was to pickle the data to a string and upload it to FileField of my model. My django model is: #models.py from django.db import models class Data(models.Model): label = models.CharField(max_length=30) file = models.FileField(upload_to="data") I...
Loading files into variables in python
I am trying to write a small function that gets a variable name, check if it exists, and if not loads it from a file (using pickle) to the global namespace. I tried using this in a file: import cPickle # # Load if neccesary # def loadfile(variable, filename): if variable not in globals(): cmd = "%s = cPickle.load(file('%s'...
Python: How do I write a list to file and then pull it back into memory (dict represented as a string convert to dict) later ?
More specific dupe of 875228—Simple data storing in Python. I have a rather large dict (6 GB) and I need to do some processing on it. I'm trying out several document clustering methods, so I need to have the whole thing in memory at once. I have other functions to run on this data, but the contents will not change. Currently, every t...
Can I pickle upload file from a django form? I mean a InMemoryUploadedFile
I have a django form where I have a FileField which accepts user's resume. I am gonna convert the resume to a html document LATER. So I thought of pickling the original document right away and store in it a db colum and later unpickle it and convert it. Is that possible? ...