Preferred way to store/retrieve python data

Question

I would like to include data files with a Python package. Is the best place to put them inside the actual package as suggested here, i.e.

setup.py
src/
    mypkg/
        __init__.py
        module.py
        data/
            tables.dat
            spoons.dat
            forks.dat

or is there a better way to do this? What is the best way to retrieve a datafile from inside python? Should I use

mypkg.__path__ + 'data/tables.dat'

for example, or should I use

pkgutil.getdata('mypkg','tables.dat')

or again, is there another better way to do this?

Generally speaking, what is the current preferred way to deal with data inside Python packages?

Answer 1

You should store your data as a Python data structure vía the Pickle module. That way, when you call it (load it) the data is ready to be used, and you dont need to process it in every script.

As for the location, it makes sense that you store it in a way that is transparent and clear to the user, the following seems intuitive to me:

from package import data

Answer 2

pkgutil means you can load the data even if the package is installed in a ZIP file, so it's preferable if you want to support that. Storing it in a data directory like that is fine, I do that all the time. :)