What are the most important infrastructure components in large-scale development?

Question

What are some of the building blocks foundational to enterprise business applications? I'm thinking about re-usable components multiple applications share to:

1. Scale up faster
2. Speed development by eliminating re-building
3. Provide a stable infrastructures by minimizing key points of failure

Some examples of concepts I've seen success with centralizing/standardizing:

• logging
• queueing
• scheduling

Answer 1

I'd add GUI widgets that allow teams to maintain the same look and feel across the application(s).

As well, in a large project where multiple teams must work on sub-systems that are going to plug together at some point a set of design patterns for similar sub-systems is invaluable.

Additionally, creating a common set of libraries (or better yet in C++ adopting Boost) which are the place to go when a common utility is required has worked well.

I would also create the testing framework with samples early on. I have found that if you have a template for testing the sub-systems then people will use it. Without any testing framework early on you are bound to get all extremes of developer testing.

Answer 2

Not completely in the infrastructure space:

• Defined logical data models.
• Defined business processes (with coverage of edge cases).

Without these you have a series of application development actives for the same customer.

• logging
• queueing
• scheduling

These should all be off the shelve to suite the needs of the applications using them. E.g. logging solution for a J2EE built application will be different to a .NET one. (In a significant enterprise mono-culture can be a problem in itself.)

Answer 3

When you say components, do you mean code components, or infrastructure components? I'd argue the infrastructure (supporting services and processes), rather than the code, gives the most benefit to quality, speed, and ease of development. The examples you give are in the standard libraries of most languages now, or are very application/use-case specific.

I'm also just going to assume version control is a given, since its so integral to all development.

I'd say the most important infrastructure would be a continuous build/automated testing. This lets people write whatever code they want, check it in, and make sure it works with everyone else's.

Second most important would be common libraries. These let people develop faster and establish standards (hopefully good) for look, feel, and design. As common libraries are used more, the importance of the continuous build increases, since small changes in those can cause regressions.

One problem with common libraries, though, is they take a lot of effort to design well - so they're worth reusing - and it takes a long time to get them to that point. Most will start as specific to a project, and then someone will hack it into their project, followed by some branching, then copy and pasting. After awhile, years probably, someone will audit the code and start merging them into a common place. A plan should be made for upgrading common libraries and how (or if) to have two versions of them at once. Another hidden drawback is they allow faster development in the short term, but hinder development in the long term, as people become locked into older versions (this is more true of third party libraries).

Third would be build tools. This means common tools to simplify building, checking out, searching, and otherwise having to interact with the code in any way. Things that automate marking bugs as resolved when you submit a patch, or notify continuous builders that dependencies have changed, or automatically running tests before you submit a test, making tests easy (and fast) to run.

Fourth would be hermetic builds, which is a simple extension of build tools. This means that an app is self contained. This has two benefits: 1) machine reuse is easier, and more importantly, 2) its incredibly easy for a developer to get started - they don't need to install extra libraries, or change something in /etc, or set something in their environment - it just builds and runs.

Some other's I can think of:

• Monitoring: If everything uses the same monitoring method, then people don't have to black-box monitor common libraries. It also makes it easier to identify problems since all the data is in one place.
• Log storage: Grepping and groking logs is a pain. Having a centralized place where all logs get stored in a structured format makes it easy to search and find problems.
• Branch management: On a large project, this lets developers develop, testers test, and releasers release without stepping on each other's toes.