pickle

How to save big "database-like" class in python

Hi there, I'm doing a project with reasonalby big DataBase. It's not a probper DB file, but a class with format as follows: DataBase.Nodes.Data=[[] for i in range(1,1000)] f.e. this DataBase is all together something like few thousands rows. Fisrt question - is the way I'm doing efficient, or is it better to use SQL, or any other "prope...

cPickle class with data save to file

Hi there, I've big class in Python it's "DataBase-like" class. I want to save it to file - all including data. This is input(example to show the issue, in script database is like 10000 records): import cPickle # DataBase-like class class DataBase: class Arrays: pass class Zones: pass class Nodes: class CR: pass ...

How to stop attributes from being pickled in Python

I am using gnosis.xml.pickle to convert an object of my own class to xml. The object is initialized so that: self.logger = MyLogger() But when I do dump the object to a string I get an exception stating that the pickler encountered an unpickleable type (thread.lock). Is there a way to 'tag' the logger attribute so that pickler will k...

load a pickle file from a zipfile

For some reason I cannot get cPickle.load to work on the file-type object returned by ZipFile.open(). If I call read() on the file-type object returned by ZipFile.open() I can use cPickle.loads though. Example .... import zipfile import cPickle # the data we want to store some_data = {1: 'one', 2: 'two', 3: 'three'} # # create a zipp...

Python Pickle: what can cause stack index out of range error?

I'm getting this error: File "C:\Python26\lib\pickle.py", line 1374, in loads return Unpickler(file).load() File "C:\Python26\lib\pickle.py", line 858, in load dispatch[key](self) File "C:\Python26\lib\pickle.py", line 1075, in load_obj k = self.marker() File "C:\Python26\lib\pickle.py", line 874, in marker while...

Python's cPickle deserialization from PHP?

Hi! I have to deserialize a dictionary in PHP that was serialized using cPickle in Python. In this specific case I probably could just regexp the wanted information, but is there a better way? Any extensions for PHP that would allow me to deserialize more natively the whole dictionary? Apparently it is serialized in Python like this: ...

How to substitute module.Class() to locally defined Class() when loading with Python's Pickle?

Hi, I have a pickle dump which has an array of foo.Bar() objects. I am trying to unpickle it, but the Bar() class definition is in the same file that's trying to unpickle, and not in the foo module. So, pickle complains that it couldn't find module foo. I tried to inject foo module doing something similar to: import imp, sys class...

Pickling Django request objects

I'm trying to pickle a request object so that I can run my view code in a debugger with valid input. However, I'm getting Can't pickle 'lock' object: <thread.lock object at 0x93ad240> I looked through the request object, but couldn't find a thread.lock object anywhere in it. Does anyone know where it is? Is there a better way to go ab...

Python encoding function can't be decoded

I wrote this python code in an attempt to convert objects to a string of ones and zeros, but the decoding fails because the data can't be unpickled. This is the code: def encode(obj): 'convert an object to ones and zeros' def tobin(str): rstr = '' for f in str: if f == "0": rstr += "0000" ...

Pickling an enum exposed by Boost.Python

Is it possible to pickle (using cPickle) an enum that has been exposed with Boost.Python? I have successfully pickled other objects using the first method described here, but none of that seems to apply for an enum type, and the objects don't seem to be pickleable by default. ...

Pickle with cucumber, machinist and Mongoid

I'm using pickle 0.3.0 with rails, cucumber and mongoid. I do found that pickle0.3.0 automatically finds the ORM. But while I wrote a cucumber scenario and tested it, It didn't find any pickle steps. Here is the sample: Scenario: logging in user Given the following users exists |name| |John| |white| But didn't find pickle ste...

Unpickling a Python class

I have a problem trying to unpickle subclasses of this class. When I unpickle it, the stuff isn't there. What gives? class Account: def __init__(self, server, port, smtp_server, smtp_port): self.server = server self.port = port self.smtp_server = smtp_server self.smtp_port = smtp_port sel...

How do I get cucumber and pickle working with mongo_mapper, machinist, and machinist_mongo?

I would like to get machinist, machinist_mongo, mongo_mapper, cucumber and pickle to play nice together. Currently I have machinist with all my blueprints configured and am using cucumber to do BDD. So far so good. My problem is I am having to write custom cucumber steps for all of my machinist blueprints. It is not really a problem per...

How to use named Machinist blueprints in Cucumber with Pickle

I noticed this little tid-bit in the readme for pickle: (The latest version of pickle supports multiple blueprints, for earlier versions of machinist use pickle <= 0.1.10) I'm running in to a situation where I want to create a user with admin privileges. I've got two machinist blueprints: User.blueprint do first_name last_name ...

Multiprocessing: using Pool.map on a function defined in a class

Hi, when i run something like from multiprocessing import Pool p = Pool(5) def f(x): return x*x p.map(f, [1,2,3]) it works fine. However, putting this as a function of a class class calculate(object): def run(self): def f(x): return x*x p = Pool() return p.map(f, [1,2,3]) cl = calculate() print cl.run() gives me ...

Django mod_wsgi PicklingError while saving object

Do you know any solution to this: [Thu Jul 08 19:15:38 2010] [error] [client 79.162.31.162] mod_wsgi (pid=3072): Exception occurred processing WSGI script '/home/www/shop/django.wsgi'., referer: http://shop.domain.com/accounts/checkout/? [Thu Jul 08 19:15:38 2010] [error] [client 79.162.31.162] Traceback (most recent call last):, refere...

Python - Using cPickle to load a previously saved pickle uses too much memory?

Python - Using cPickle to load a previously saved pickle uses too much memory? My pickle file is about 340MB but takes up 29% of 6gb of memory when loaded. This seems a bit too much. The pickle file is a dictionary of dictionaries. Is this appropriate? Code used: import cPickle as pickle file = pickle.load( file_handle ) Thanks ...

How to acces a model attribute using Pickle in a cucumber test

Hello everyone ! I am currently trying to use the Pickle gem for my cucumber test in ruby and rails. I use blueprints. If I have a User class, with an id as primary key, and a "name" and a "best_friend_id" as attributes. How can I access to the id of the first User "Toto" to fill the best_friend_id field of "Titi" in my Pickle steps? F...

Can't pickle loggers??

Working on a project that requires that I am able to pickle the container object at any point, since we expect it to fail on external conditions quite frequently and be able to statefully pick up where we left off. I'm using the python logging library quite extensively, and all of my classes start by setting up a logger like: class foo...

Python Pickle Help

I'm not sure why this Pickle example is not showing both of the dictionary definitions. As I understand it, "ab+" should mean that the pickle.dat file is being appended to and can be read from. I'm new to the whole pickle concept, but the tutorials on the net don't seem to go beyond just the initial storage. import cPickle as pickle de...