Python: Read configuration file with multiple lines per key

Question

Hi,

I am writing a small DB test suite, which reads configuration files with queries and expected results, e.g.:

query         = "SELECT * from cities WHERE name='Unknown';"
count         = 0
level         = 1
name          = "Check for cities whose name should be null"
suggested_fix = "UPDATE cities SET name=NULL WHERE name='Unknown';"

This works well; I divide each line using Python's string.partition('=').

My problem is very long SQL queries. Currently, I just paste these queries as a one-liner, which is ugly and unmaintainable.

I want to find an elegant, Pythonic way to read the right of an expression, even if spans over many lines.

Notes:

• my SQL queries might contain the =
• I don't fancy the idea of forcing "s around the right hand side, because there are many existing files without it.

EDIT:

ConfigParser is great, but it forces me to add a space or tab at the beginning of every line in a multiline entry. This might be a great pain.

Thanks in advance,

Adam

Answer 1

configparser

Answer 2

ConfigObj

Answer 3

The Python standard library module ConfigParser supports this by default. The configuration file has to be in a standard format:

[Long Section]
short: this is a normal line
long: this value continues
in the next line

The configuration file above could be read with the following code:

import ConfigParser
config = ConfigParser.ConfigParser()
config.read('longsections.cfg')
long = config.get('Long Section', 'long')

Answer 4

I would you suggest to use a regular expression... The code might look like this to give you are start:

import re

test="""query = "select * from cities;"
count = 0
multine_query = "select *
from cities
     where name='unknown';"
"""

re_config = re.compile(r'^(\w+)\s*=\s*((?:".[^"]*")|(?:\d+))$', re.M)
for key, value in re_config.findall(test):
    if value.startswith('"'):
        value = value[1:-1]
    else:
        value = int(value)
    print key, '=', repr(value)

The output of this example is:

~> python test.py 
query = 'select * from cities;'
count = 0
multine_query = "select *\nfrom cities\n     where name='unknown';"

Hope that helps!

Regards, Christoph

Answer 5

This is almost exactly the use-case that made us switch to YAML (Wikipedia, python implementation, documentation; you might want to look at JSON as an alternative). YAML has some advantages over configparser or json:

• human readability (better than JSON for larger files);
• can serialize arbitrary python objects (which makes it as un-safe as pickle, but there is a safe_load function in the python implementation to alleviate this issue). This is already useful for something as simple as a datetime object.

For completeness sake, the main disadvantages (IMO):

• Python implementation by an order of magnitude slower than JSON implementation;
• less portable across platforms than JSON.

For example

import yaml

sql = """
query         : "SELECT * from cities
WHERE name='Unknown';"
count         : 0
level         : 1
name          : "Check for cities whose name should be null"
suggested_fix : "UPDATE cities SET name=NULL WHERE name='Unknown';"
"""

sql_dict = yaml.safe_load(sql)

print(sql_dict['query'])

prints

SELECT * from cities WHERE name='Unknown';