How can I build a generic dataset-handling Perl library?

Question

Hello,

I want to build a generic Perl module for handling and analysing biomedical character separated datasets and which can, most certain, be used on any kind of datasets that contain a mixture of categorical (A,B,C,..) and continuous (1.2,3,881..) and identifier (XXX1,XXX2...). The plan is to have people initialize the module and then use some arguments to point to the data file(s), the place were the analysis reports should be placed and the structure of the data.

By structure of data I mean which variable is in which place and its name/type. And this is where I need some enlightenment. I am baffled how to do this in a clean way. Obviously, having people create a simple schema file, be it XML or some other format would be the cleanest but maybe not all people enjoy doing something like this.

The solutions I can think of are:

• Create a configuration file in XML or similar and with a prespecified format.
• Pass the information during initialization of the module.
• Use the first row of the data as headers and try to guess types (ouch)

Surely there must be a "canonical" way of doing this that is also usable and efficient.

Thanks p.

Answer 1

This doesn't answer your question directly, but have you checked CPAN? It might have the module you need already. If not, it might have similar modules -- related either to biomedical data or simply to delimited data handling -- that you can mine for good ideas, both concerning formats for metadata and your module's API.

Answer 2

Any of the approaches you've listed could make sense. It all depends on how complex the data structures and their definitions are. What will make something like this useful to people is whether it saves them time and effort. So, your decision will have to be answered based on what approach will best satisfy the need to make:

• use of the module easy
• reuse of data definitions easy
• the data definition language sufficiently expressive to describe all known use cases
• the data definition language sufficiently simple that an infrequent user can spend minimal time with the docs before getting real work done.

For example, if I just need to enter the names of the columns and their types (and there are only 4 well defined types), doing this each time in a script isn't too bad. Unless I have 350 columns to deal with in every file.

However, if large, complicated structure definitions are common, then a more modular reuse oriented approach is better.

If your data description language is difficult to work with, you can mitigate the issue a bit by providing a configuration tool that allows one to create and edit data schemes.

Answer 3

rx might be worth looking at, as well as the Data::Rx module on the CPAN. It provides schema checking for JSON, but there is nothing inherent in the model that makes it JSON-only.