Using numpy's flatten_dtype with structured dtypes that have titles

Question

Hi,

I usually don't post questions on these forums, but I've searched all over the place and I haven't found anything about this issue.

I am working with structured arrays to store experimental data. I'm using titles to store information about my fields, in this case the units of measure. When I call numpy.lib.io.flatten_dtype() on my dtype, I get:

ValueError: too many values to unpack  
File "c:\Python25\Lib\site-packages\numpy\lib\_iotools.py", line 78, in flatten_dtype
  (typ, _) = ndtype.fields[field]

I wouldn't really care, except that numpy.genfromtxt() calls numpy.lib.io.flatten_dtype(), and I need to be able to import my data from text files.

I'm wondering if I've done something wrong. Is flatten_dtype() not meant to support titles? Is there a work-around for genfromtxt()?

Here's a snippet of my code:

import numpy
fname = "C:\\Somefile.txt"
dtype = numpy.dtype([(("Amps","Current"),"f8"),(("Volts","Voltage"),"f8")])
myarray = numpy.genfromtxt(fname,dtype)

Answer 1

Here is a possible workaround:

Since your custom dtype causes a problem, supply a flattened dtype instead:

In [77]: arr=np.genfromtxt('a',dtype='f8,f8')

In [78]: arr
Out[78]: 
array([(1.0, 2.0), (3.0, 4.0)], 
      dtype=[('f0', '<f8'), ('f1', '<f8')])

Then use astype to convert to your desired dtype:

In [79]: arr=np.genfromtxt('a',dtype='f8,f8').astype(dtype)

In [80]: arr
Out[80]: 
array([(1.0, 2.0), (3.0, 4.0)], 
      dtype=[(('Amps', 'Current'), '<f8'), (('Volts', 'Voltage'), '<f8')])

Edit: Another alternative is to monkey-patch numpy.lib.io.flatten_dtype:

import numpy
import numpy.lib.io
def flatten_dtype(ndtype):
    """
    Unpack a structured data-type.

    """
    names = ndtype.names
    if names is None:
        return [ndtype]
    else:
        types = []
        for field in names:
            typ_fields = ndtype.fields[field]
            flat_dt = flatten_dtype(typ_fields[0])
            types.extend(flat_dt)
        return types
numpy.lib.io.flatten_dtype=flatten_dtype