What are the CS fundamentals behind package/dependency management?

Question

Often I hear about situations where companies are developing extensable in house software (the dreaded enterprise 'framework') which is supposed to support multiple 'plugins' from diffirent teams. Usually this ends up being a half baked solution that does not really work due to compatibility prolems between addins, or between addins and the framework itself. Usually this means QA have to 'rubber stamp' a global set of versions accross all plugins, or more usually plugins are released and stuff breaks in nasty ways.

This problem has been solved before however, for example the package management systems like apt for debian linux. I suspect that the reason it works is that it is built from the start on a known 'Computer Science-y' concept. My question is what is it?

Answer 1

I don't know of any CS basis, but I have found ZeroInstall to be an intriguing alternative to mainstream package management models. It allows multiple versions of software to co-exist with optimal sharing of package dependencies. Roughly speaking, you run apps directly from the web, with a cache storing all the bits needed to run locally. Strong hashing and versioning ensures reliable caching, so that you don't find yourself running good code one day, and rubbish (or a virus) the next.

Answer 2

I don't think there is any computer science concept behind it. If you have ever seen complex systems, you could probably say so.

I work for a company with a few thousands developers, solely for in-house software (we've got a few servers as you can imagine). Coordinating the developments is quite a colossal task of course.

The main point is to realize that there are various ways of communicating:

• the obvious way: DLL / Plugin / you name it
• classical messaging way: over tcp/ip etc...
• shared files

This can be separated into 2 categories:

• structured
• unstructured

Unfortunately, for ease of development, many rely on unstructured things: I pass a string which contains the name of Surname followed by a slash, followed by GivenName... and the day you add Title or MiddleName everything goes awry.

Of course, if you had proper testing this would not fail, but let's face it, we always forget one path or another, and thus structured communication help. This is achieved, when communicating by files / network, by good messaging libraries that are shared among all:

• strong versioning of the messages format
• backward compatibility because not everyone migrate at the same time

It works, I witness it everyday, but it takes some willpower to impose it on the laziest developers around who find that designing a structured class is a loss of time... fortunately, you can focus the less lazy ones on the framework, which is probably the one piece you don't want to fail.

Answer 3

I don't know of a known algorithm, but I do know how we deal with it.

The first step is to promote strong versioning of the different components. Once a component is delivered, it cannot change, ever. A fix for a component means delivering a new version. This is traditionally achieved through 3 groups of digits 1.2.3.

Now, each component should explicitly reference its own dependencies.

[
  "name": "Foo",
  "version": "1.2.3",
  "dependencies": [ "Bar": "1.3.5", "Xtr": "5.3.4" ]
]

You can thus build a dependency graph and explore it:

• looking for cycles (uh, bad)
• checking that versions of dependencies are aligned (the same library should not appear with not-binary compatible versions)
• etc...

Of course, you could use xml instead of json...

We have something similar where I work, though a bit more complicated because it's also use as a basis for our build system (which generates makefiles):

• versioner system: this way you don't precise the version of the component explicitly, you reuse the one defined by the versioner, which may itself use a versioner etc...
• configuration system: each version can be declined for a number of configurations (for example mysql vs ora vs sqlite)

The versioner is a good way to align the various versions. The configuration is a bit more complicated and you'd better not deal with it if you don't have to.