Dealing with complexity

Question

Forgive me for asking a rather general question, but how do you psychologically deal with high levels of complexity in software projects?

I'm working on a personal compiler project and am currently trying to understand the unmanaged .net metadata API (and before anybody mentions it, Reflection.Emit is not sufficient for my needs!) which are horribly opaque and have very little documentation. Half the time the documentation consists of PaintFooRed() - 'this function paints Foo red.' so I'm really scrabbling around in the dark trying to work things out through trial, error and working things out from first principals.

When dealing with enormous complexity like this I find the uncertainty of a. how long something may take, b. whether it's even possible and c. whether the approach I'm taking is correct is very difficult to deal with.

How do you deal with this kind of situation?

I've found that solving problems of this degree of uncertainty and complexity are a considerably greater joy than finding the solution to more trivial tasks - has anybody else experienced this too?

Answer 1

I try breaking giant projects down into smaller modular chunks. Then you can try to manage the smaller parts.

I know that is a very vague answer. But it is the best I can come up with.

Answer 2

I've had this come up several times in my career.

• When I'm trying to understand poorly documented or undocumented APIs, I usually start by writing small little test apps (for vendor APIs) or unit tests (for APIs within my project).
• For complex problems, if I can break down what needs to be done into the smallest pieces possible, it makes it easier to wrap my head around it.

• If I'm trying to understand legacy code, and it's code I have write access to, I will add my own documentation to the code as I come to a better understanding of it.

• Failing that, I will take notes, make diagrams (I'm a picture person, pictures help me a lot), etc. either on paper or electronically, to help me make and remember the mental connections that I discover between the various functions, concepts, or whatever.

Answer 3

a. Other than relying on gut feel and experience, there is no easy way to judge how long such a "walk in the dark" will take. The best you can do is to try and break the task down into lots of small steps and record how long each step takes. As time goes by, the average length of time for each step will act as a crude metric for how long the remaining steps will take (on average)

b. Many things will seem impossible unless you are stubborn enough. If you are too stubborn though, you could waste days or weeks on something that really is impossible. Discuss what you are doing with other like-minded people is the best here. If they all say give up, then listen to them. If not, push on if you are happy to take the risk.

c.There is rarely a correct way to do something. Weirdly though, there are a great many wrong ways to do things. Again, talking to others is your best bet here. Try to get a consensus on the est way and take that.

Answer 4

Psychologically I find the most important thing is no distractions. Its relatively easy to code around complex problems, but once your focus is broken (usually by other work, bugfixes and so on) then I find it takes a while to get back into the mindset I was in when working on the complex problem.

When working on such things, things tend to fit into compartments in my memory - so I suppose modularisation is very important to keeping track of everything.

Answer 5

If one is available, having a mentor would be a great way to deal with complexity. The psychological benefit is that there is a "safety net" that you could turn to in the event that you are unable to figure out how something works.

The challenge would be to make sure that you are learning from the mentor and not using them as a crutch.

Answer 6

Use paper or a white board. Use words, diagrams and mind maps to describe the system and slowly decompose the system into subsystems. Understand each subsystem. Try to define interfaces between subsystems.

Try to follow one transaction through the whole system. How does an "if" statement get compiled? Work it through from start to finish and simulate the computer. Be the computer and execute the work.

Try to explain the process to someone else. Having to verbalize it and see how confused people look will help your understanding.

Find metaphors if you can and give names to things that may not have names. This will help form sentences that will help you remember what is going on.

Answer 7

I found out that a good academic background helps you to solve prolems that otherwise you would end up reinventing. I also found out that extensive and detailed testing (of course, it depends on the complexity of what you are developing, among other factors) helps you gain confidence in your app.

Answer 8

The classic engineering approach is to hierarchically define the system into layers, using black-box concepts and abstractions to move the complexity into a level you can personally manage. At the ground level of your system, when you are building isolated components, you should test them thoroughly, as that will permit you to install them and trust that they will be solid.

You also need to define milestones that you can throw into a test harness and verify that yes, you ARE doing what you are supposed to be doing at that point in time.

This is not ad hoc hacking. This is software engineering, and doing it right requires putting forth a lot of regular design time and effort - if you want a quality product.

Answer 9

When the documentation is bad, or if you only have a reference doc, no tutorials or any "start here" point, what i do is:

• read it all, from start to finish. if you don't understand something, reread this part once or twice, don't fight it. the objective is to read all

• sleep. if it was small, just take a walk thinking of something else. if it was big (over 500-1000 pages) the whole night.

• pick a subject, anyone, that you think was (mostly) clear from the doc, and read it again. note how you can put in context of the whole thing that is slowly taking form in your mind.

• repeat for a few subjects, trying to gently walk your way to basic concepts, until you find something that can be explained using only what you knew before reading. this is your "start point"

• now, and only now, begin experimenting. start with your startpoint, and gradually add more concepts, use the reference doc for this. even if it's incomplete, you already know your way there.

Answer 10

Your question is along the lines of things I ponder often. To me complexity in software in a bit like the surface area to volume ratio of a sphere. As you add features/lines of code etc., a modest increase in surface area, you increase the complexity(volume) by the cube. The only measures you can take to combat this is to effectively modularize. The same is true for biological systems and complex organisms.

In code I find that things should generally get simpler as you break down responsibilites. If not, then my instincts are that I have designed something wrong.

When you are in completely uncharted territory and deciding whether or not something is even possible the trial and error tricks of the alchemists come in handy. (It also comes in handy to be paid by the hour)

While I have never built a compiler, I have developed mini languages in both Java using JavaCC and in .NET using Gold Parser and Antlr. In each case the there was a very high degree of uncertainty and complexity, but, the sense of accomplishment is very high indeed.

Answer 11

Complexity, by definition means composed of many parts. To understand something complex, you have to decompose it into parts that you can work with and reason about.

The biggest source of input a brain gets is visual data about the physical space around us. So we're inherently good at dealing with things laid out with the relationships represented pysically. So I would suggest drawing diagrams.

To understand an undocumented API though... the only options are trial and error, to track down the author, or ask someone who's already spent time with one of these.

You could ask specific questions about the API here, and see if you can get some help from somebody!

Answer 12

Get a copy of Mythical Man Month and read up on Accidental versus Essential Complexity

Answer 13

I find it helpful to use a combination of techniques. Interestingly the process of analyzing a new technology is almost identical, at first, to the process of using it to create: both start with a very general understanding of the whole, then progress to the details. The one major difference occurs about 3 steps into the process: analysis usually involves moving from the abstract to the concrete, while creating usually progresses from the concrete to the abstract.

• Start by building a high-level understanding. It's easier to fit details into an overall context than to assimilate a lot of seemingly independent information. Additionally, if you can see the "big picture," it's sometimes easier to recognize pieces that are missing.

• If it's possible, start with a small example. It's easier to analyze a working model than to create from scratch. You'll get an understanding of relevant techniques, idioms, and implementation details that will clarify your context and provide templates for your own work.

• Move from the abstract to the concrete. Choose one concept at a time and explore it. What is its purpose? What are its components? Don't go to deep if you're concerned about being overwhelmed; just try to understand how it relates to the overall context.

• Drill into the details. As you progress from analysis to creation, you'll need to understand an increasing amount of detail. Depending on your own style, you may prefer to stay in one area at a time and increase your "radius" of expertise, or you may prefer a "shotgun" approach that gives you detailed information about many different parts of the whole. In either case, you're building knowledge of the fundamentals and increasing your overall understanding and confidence.

• Explain it to someone else. The process of articulating the information forces you to organize it in your own mind, and it highlights the areas you do and don't understand well. You'll also identify questions and synthesize new ideas and connections.

• Create something! Whether it's the framework for a "real" project or a throw-away experiment, start using the technology. There's no substitute for total immersion, and you'll get satisfaction from building working examples.

  Good luck!