Where\How to store distributed configuration data

Question

A single installation of our product stores it's configuration in a set of database tables.

None of the installations 'know' about any other installation.

It's always been common for customers to install multiple copies of our product in different datacentres, which are geographically far apart. This means that the configuration information needs to be created once, then exported to other systems. Some of the config is then modified to suit local conditions. e.g. changing IP addresses, etc. This is a clunky, error prone approach.

We're now getting requests for the ability to have a more seamless strategy for sharing of global data, but still allowing local modifications.

If it weren't for the local modifications bit then we could use Oracle's data replication features.

Due to HA requirements having all the configuration in the one database isn't an option.

Has anyone else encountered this problem and have you ever figured out a good programmatic solution for this? Know of any good papers that might describe a partial or full solution?

We're *nix based, and use Oracle. Changes should be replicated to all nodes pretty quickly (a second or 2).

Answer 1

I'm not sure how possible it is for you to change the way you handle your configuration, but we implemented something similar to this by using the idea of local overrides. Specifically, you have two configuration tables that are identical (call them CentralConfig and LocalConfig). CentralConfig is maintained at the central location, and is replicated out to your satellite locations, where it is read-only. LocalConfig can be set up at the local site. Your procedure which queries configuration data first looks for the data in the LocalConfig table, and if not found, retrieves it from the CentralConfig table.

For example, if you were trying to do this with the values in the v$parameter table, you could query your configuration using the FIRST_VALUE function in SQL analytics:

  SELECT DISTINCT
         NAME
       , FIRST_VALUE(VALUE) OVER(PARTITION BY NAME 
                                     ORDER BY localsort
                                ) VALUE
    FROM (SELECT t.*
               , 0 localsort
            FROM local_parameter t
           UNION
          SELECT t.*
               , 1 localsort
            FROM v$parameter t
         )
ORDER BY NAME;

The localsort column in the unions is there just to make sure that the local_parameter values take precedence over the v$parameter values.

In our system, it's actually much more sophisticated than this. In addition to the "name" for the parameter you're looking up, we also have a "context" column that describes the context we are looking for. For example, we might have a parameter "timeout" that is set centrally, but even locally, we have multiple components that use this value. They may all be the same, but we may also want to configure them differently. So when the tool looks up the "timeout" value, it also constrains by scope. In the configuration itself, we may use wildcards when we define what we want for scope, such as:

CONTEXT       NAME    VALUE
------------- ------- -----
Comp Engine A timeout    15
Comp Engine B timeout    10
Comp Engine % timeout     5
%             timeout    30

The configuration above says, for all components, use a timeout of 30, but for Comp Engines of any type, use a timeout of 5, however for Comp Engines A & B, use 15 & 10 respectively. The last two configurations may be maintained in CentralConfig, but the other two may be maintained in LocalConfig, and you would resolve the settings this way:

  SELECT DISTINCT
         NAME
       , FIRST_VALUE(VALUE) OVER(PARTITION BY NAME 
                                     ORDER BY (TRANSLATE(Context
                                                        , '%_'
                                                        , CHR(1) || CHR(2)
                                              ) DESC
                                            , localsort
                                ) VALUE
    FROM (SELECT t.*
               , 0 localsort
            FROM LocalConfig t
           WHERE 'Comp Engine A' LIKE Context
           UNION
          SELECT t.*
               , 1 localsort
            FROM CentralConfig t
           WHERE 'Comp Engine A' LIKE Context
         )
ORDER BY NAME;

It's basically the same query, except that I'm inserting that TRANSLATE expression before my localsort and I'm constraining on Context. What it's doing is converting the % and _ characters to chr(1) & chr(2), which will make them sort after alphanumeric characters in the descending sort. In this way, the explicitly defined "Comp Engine A" will come before "Comp Engine %", which in turn will come before "%". In cases where the contexts are defined identically, local config takes precedence over central ones; if you wanted local to always trump central, even in cases when central was scoped more tightly, you'd just reverse the positions of the two sort terms.