How can I have super-fast dynamic configuration?

views:

100

answers:

+3 Q:

How can I have super-fast dynamic configuration?

disclaimer: I work on a tool to assist deployment (but I'm not advocating it's usage for you)

I suggest you just have a private website that distributes the configs, and simply cache them in your 'reading' apps. Then you may have the ability for the central server to 'ping' the externals, when an update occurs, suggesting that they refresh their cache (or have that happen as part of a cache expiry on the clients).

Noon Silk 2009-09-21 12:09:21

Good tip; I'm not sure it would work in this case cause we're using a common code-based in the application servers which could have hits coming in on one domain at one moment, and then another domain (deployment) the next moment.. they kinda can't remember their context/configs unfortunately.

Geoff McQueen 2009-09-21 13:47:16

Geoff: Then you key your cache on that basis.

Noon Silk 2009-09-21 14:05:31

So, the idea would be to store all the config variables in memory, and then access them at run-time that way? Can't argue with the 'speed' benefits of this approach... isolating namespace aspects might be a bit of fun.

Geoff McQueen 2009-09-21 14:44:39

Indeed; You can store them where you want; it could be memcached, it could be a file, it could be a DB (as someone else suggests); the main trick is to get them from a central location, and only update from _that_, based on expiry.

Noon Silk 2009-09-21 14:51:36

+1 A:

My solution was to put the config in the same database as the application. That way, I could simply pass one DB connector to the app and it would use the correct config.

In the app, the config was accessed via a global config instance which would read all values from the database in one go and cache them. For a web based app, I use a special URL to tell the config instance to refresh itself.

For other apps, I use a local file. When the file exists, the config data is reread. You can do this in a second thread or just check for the file every time. Since the path is static, the OS can optimize this access until it take very little time. After the config has been read, the file is deleted, so I know that the reload happened.

Aaron Digulla 2009-09-21 13:36:36

One of the config elements we need to draw in is what database name to connect to... while putting every deployment into one big DB is possible, it has some clients a bit worried and so having discrete DB's and 'partitioning' them provides a bit of natural sharding as well as a sense of added security/privacy for clients.I guess my other question is whether something like MySQL using InnoDB or MyISAM would be as 'quick' in a read heavy environment as some other database types...

Geoff McQueen 2009-09-21 13:49:12

My suggestion is to give the DB name as the sole config element to the app. So you can have many DBs and each contains a copy of the config for itself. Example: You have DBs "dev", "test", "prod". Same app, each gets the name as startup argument and then reads the correct config from the right DB.

Aaron Digulla 2009-09-21 13:54:35

This may be a "non answer" of sorts, but consider the following: are you trying to solve a perceived problem in a way that would introduce even more issues in the future? You've mentioned your reliance on Perl data structures that live inside config files. Consider for a moment that all this configuration data lives in a RDBMS. How would you go about replicating the functionality that you get "for free" with your existing approach, in a system that provides a very different set of functionality? Wouldn't you at some point end up with CLOB columns containing your good old Perl data structures?

What exactly is driving you away from your current configuration mechanism? I have a feeling that those issues could be addressed on their own without necessitating a large-scale architectural upheaval.

Max A. 2009-09-21 14:23:21

We're not using too much in the way of structures; we could go back to key/value pairs and through naming them 'slurp' them into the Perl data structures we already have if we need to. We just need to get to a point where we are able to deploy new deployments/domains without touching any .pl or static configs. Load balancing as well as optimizing a core code set are the drivers.Still, your warning about thinking through consequences and testing non-existent concerns is appreciated... best not to go for overkill.

Geoff McQueen 2009-09-21 14:39:40

ansaurus

tags:

views:

answers:

How can I have super-fast dynamic configuration?

related questions