What are some ways to test an architecture?

Question

How does one test an architecture? Are there test-related things that can be done while the architecture is still being pounded into shape?

I plan to iterate but I don't want to wait to have an entire architecture done (even if it's very rough) before I start thinking about testing.

Update: I'm using the word architecture the way it's used in Code Complete.

In other words, at this second, my architecture is a pile of paper showing how various blobs interact (e.g. one sheet of paper shows a legacy system that talks to a facade that parses/slices-and-dices, which in turn talks to the main website application).

Later, I'll flesh it out to the point of public classes/methods. And after that, I'll write out private method signatures and poke at algorithms, though I'm already looking into them for high-level things such as classifying text.

Answer 1

If you want to iterate you should only design that much that you need for the iteration. Architecture is mostly seen as one layer higher than design. That means I use a layered MVC architecture for deciding how to design my class structure.

If you are building an overall program structure for the first iterations without adding features and things that can be tested you are doing something wrong. One of the benefits from going forward with an iterative process is that you don't over design that much in directions you don't need it.

Take the program that you want to build and take the three most important features. Think about how to implement them and how to design the class structure for that. Think about the things that are likely to change and make them changeable in your Design. Then start your first iteration. Before starting your first iteration think about the architectural choices, are you using MVC, a client server based architecture, do you have a Database Layer or something comparable. Those are choices that will have influence of your later design choices.

For further reading I suggest http://en.wikipedia.org/wiki/Iterative_and_incremental_development and Planning XP-Programming from Kent Beck.

Answer 2

A SOFTWARE ARCHITECTURE-BASED TESTING TECHNIQUE

This dissertation presents a practical, effective, and automatable technique for testing architecture relations at the architecture level. It also presents a proof-of-concept tool to generate test requirements.

http://cs.gmu.edu/~offutt/rsrch/JinDiss.pdf

Answer 3

Usually, "testing an architecture" means testing an applicative and technical architecture (as opposed to business and functional one, which is not so much "tested" as "validated").
And this is not "unit testing" or "continuous integration" or TDD either, but more a black-box approach for testing a all system (composed from many modules) from start to finish.

You can indeed start designing not one, but several testing policies: when they concern "an architecture", they are often "system-wide" tests that implies a dedicated infrastructure (network, server close to the deployment target).

So, when designing your first "architectural" tests, you can take into account:

• what kind of test you want to make first (regression, stress-tests, uat -- user acceptance tests --)?
• what kind of traceability you want between those tests and the initial set of requirements?
• what kind of deployment process you need to automate in order to easily and quickly set up or update a testing platform (especially if there is a base involved, how will you deal with schema upgrade, and testing data?).

All of those topics can be addressed while the architecture is refined, in order to support more efficiently the development effort as soon as it begins.

Answer 4

Your archecture is a design document right?

Inspections are what you will want.

Inspections can catch bugs just as well as testing. Inspections will catch "bugs" sooner.

Put the documents in front of a group of knowledgeable people. Ask them to read it and to tell you where there are issues. Rework your document and get it back to them. Continue this process until it is completely clear to your knowledgeable audience what you are doing and all are convinced it will work.

Hopefully your document is written in a way where your readers can understand the interactions and see the workings even though they are abstract.

Answer 5

ATAM is something you should look at. It's a method for evaluation architectures. A key step is coming up with "scenarios", basically things the system might be required to do in the future, changing or expanded requirements. For example, having the system run on a different OS. You then evaluate how the architecture would accomodate these scenarions.

Answer 6

Design Rule #1: Don't assume anything is easy, thinking a system will be easy usually means the requirements or scope of what has to be coded hasn't been thought through yet.

Design Rule #2: Don't design something and then hand it off to developers and tell them it's easy when you haven't actually coded or worked with the technology yet.

Design Rule #3: New technology is not a silver bullet that makes all of your problems go away. All new technologies have integration challenges and often many bugs that have yet to be fixed by the vendor.

Bearing these rules in mind, proceed as follows:

Define the technical assumptions, unknowns, and new technologies being introduced or considered.

Validate any unproven assumptions - i.e. something you or the team has not worked with before. Reading a white paper or vendor press release doesn't count. Make sure it works before building your system on it if its something that hasn't been done before or is being used in new way.

Build a hardcoded throw-away prototype to verify system interactions and unknowns can be solved with a minimum of workarounds. Drop any part of your plan that isn't working and find another way to do it before it becomes a black hole that consumes your architecture.