How do you fight design complexity?

Question

I often find myself fighting overengineering -- the person in charge of designing the software comes up with an architecture that's, way, way overcomplicated.

It's all fine and dandy to have all the esoteric features that users will never know about and get that sense of achievement when you're doing something that all the magazine articles are telling you is the latest, cool thing, but we are going to spend half of our engineering time on this monument to our cleverness, and not, you know, the actual product that our users need and upper management expects to be completed within a reasonable or at least a bounded time frame.

And you'll probably just have to revert back to the simpler solution anyway when you start running out of time, that is, if you get that chance.

We've all heard the refrain: Keep It Simple, Stupid™.

How do you fight with overcomplexity in your team?

---

One example I've had to work with repeatedly lately is when the decision has been made to go to a fully denormalized database design rather than an RDBMS. "because it's faster!" Fully denormalized databases are really hard to get right, and are only appropriate for really specialized data problems like Flickr or ebay, and which can be extremely expensive in terms of developer time relative to the rest of your development.

Answer 1

Tooth and nail, and sometimes you lose. The problem is that it's always easy to be tempted to build something cool.

Why build something simple and efficient when it can be complicated and wonderful?

Try to remind people of the XP rule of building the simplest thing that can possibly work.

Answer 2

You may suffer from "too many architects in the team" syndrome. One or two people at most should design/architect a system which will be coded by a team of 5 to 10 people. Input is welcome from everyone but architectural decision makers should be few and experienced.

(the numbers are semi-random and could be different depending on other factors as well)

Answer 3

I try to be open when discussing matters. But when i am discussing with someone else between something that seems simple and another one complicated, i get as stubborn as can be. It helps quite a lot, so long as you are very coherent from one decision to another.

Answer 4

I assume you mean "fully denormalized database design rather than a normalized (e.g., third or fourth normal form) model", because a relational model is managed by an RDBMS regardless of how normalized it is.

Can't judge without knowing more about your requirements, your abilities, and those of your teammates.

I fear that your KISS admonition might not work in this case, because one big, denormalized table might be defended as the simplest thing possible.

How does anybody solve these kinds of problems? Communication, persuasion, better data, prototypes of alternative technologies and techniques. This is what makes software development so hard. If there was only one way to do these things, and everyone agreed on them, we truly could script it or get anyone to develop systems and be done with it.

Get some data. Your words might not be enough. If a quick prototype can demonstrate your point, create it.

"Strong opinions, lightly held" should be your motto.

Sometimes a technical point isn't worth alienating your entire team. Sometimes it is. Your call.

Answer 5

At least for me, the bigger issue is that it's often hard to tell what feature is in there because of its buzzword-friendly, magaziney enterprisey goodness and which is in there because it adds a level of flexibility that will be useful in the future.

It's been shown that people are generally terrible at anticipating future complexity, and side-effects of current decisions. Unfortunately this doesn't always mean simplest is best - in my case, there've been loads of things I thought were too complicated at first and didn't see the value of until much later (er... spring). Also things I thought made sense that turned out to be wildly overcomplicated (EJB1). So I know that my intuition about these things is faulty.

Best bet - any kind of indirection layer should be supported with an argument supporting the value of the flexibility it adds vs. its added dev complexity.

However, people who are dogmatically maintaining a particular db setup on abstract grounds are probably in the "building it because I read that it's the right thing" camp. It might be unrealistic, but some people might be convinced if you build a test version and benchmark, especially if the results show more effort leading to an insignificant performance increase.

Answer 6

It's all fine and dandy to have all the esoteric features that users will never know about and...

This would be feature creep, not unnecessarily complicated design. It's different from your example on databases.

One example I've had to work with repeatedly lately is when the decision has been made to go to a fully denormalized database design rather than an RDBMS. "because it's faster!"

In this case several things may be going on. One of them is, you might be wrong and these people could really know what they are saying because they have worked with very similar examples. Another is they might be wrong, i.e. their design doesn't offer the speed advantages they claim. In this case there could be two different situations: (1) They are giving speed too much weight in their design, or (2) speed is really critical. If speed is indeed so relevant, the team shouldn't rely only in assumptions - they should try different prototypes and evaluate their speed in the critical paths. You don't build an F1 car in one way just "because it's faster", instead you keep trying several alternative design solutions and pick the fastest one which still doesn't increase maintenance costs too much.

Sometimes you can argue it and reach an agreement, sometimes you can't. It's life.

A final word, though. You don't fight complexity. You treat it. You identify the really important things and act accordingly.

Answer 7

Your example isn't really a complicated design, it's a design choice that you don't agree with. Since you're working on the code, you could easily be right because many of these decisions are made by people reading an article in an article and thinking it sounds like a good goal, or the decision could have been made by someone who ran into problems before and was trying to prevent it from happening again.

Personally I've done a lot of stuff the easy way and a lot of it the hard way, and I'm never happy when I choose doing something the easy way over the hard way. Now I've learned tricks like "never pass around a naked collection, always wrap it in a business class".

If I were to explain the reasoning behind that to someone who hadn't been through the same experiences, they wouldn't understand it until they tried comparing the "easy way" to the "hard way" a few times.

Answer 8

The solution should be no more complex than the problem.

The question intertwines itself with the thought of essential complexity. A sort must touch each element, by its essence. How much more complex must it then get, to solve the problem, given the technical constraints existing on it?

Answer 9

Do the people involved have enough time and incentive to find a simple solution? Without care, complexity will increase. If you spend most of your time trying to do the quickest possible bug fix or feature addition then saying "keep it simple" will not be enough.

Ensure that there are some people on the team with war wounds from maintaining large programs, and people with experience of refactoring, and then give them time to sort the software out. If you can arrange that certain features and opportunities are out of scope, that will help people remove unneeded code. If you want a metric, aim to reduce lines of code; but try not to obsess over it. Improve the test coverage; consider eliminating bits of code which are hard to test.

Answer 10

Don't try to do everything at a stretch. Break every problem/task into manageable chunks. Then prioritize, keeping KISS and YAGNI in mind. This will help you focus on building what you need. If you've done it right, you'll have a good core you can add to later, given time, money, resources and inspiration.

Answer 11

You fight someone else's overdesign/feature creep in several ways:

1. Request feature priority, based on actual user requirements. Mock up features for alpha and beta testers, and ask if they would trade N months of delay for it.

2. Aggressively refactor to avoid special-casing. Break code into layers or modular components whenever appropriate. Find a balance between "works just fine now" and "easy to extend later".

3. Notify your management when you disagree with design decisions, prepare to be overruled, and accept the decision. Don't go over anyone's head or sabotage code.

Answer 12

Make sure to bounce any ideas you have off of someone else. Oftentimes, we get so wrapped up in doing things a certain way that it takes another set of eyes to set you right. There've been many times that I've figured out difficult problems by having somebody else there to say "do we really need that?" This helps make my code simpler.

On the point of dealing with people you disagree with, Ward Cunningham has a good point:

It was a turning point in my programming career when I realized that I didn't have to win every argument. I'd be talking about code with someone, and I'd say, "I think the best way to do it is A." And they'd say, "I think the best way to do it is B. I'd say, "Well no, it's really A." And they'd say, "Well, we want to do B." It was a turning point for me when I could say, "Fine. Do B. It's not going to hurt us that much if I'm wrong. It's not going to hurt us that much if I'm right and you do B, because, we can correct mistakes. So lets find out if it's a mistake. ... Usually it turns out to be C. It's a learning experience for both of us. If we decide without the experience, neither of us really learns. Ward won, and somebody else didn't. Or vice versa. It's too much of a battle. Why not say, "Well, let's just code it up and see what happens. If it doesn't work, we'll change it.""

My advice? If you want to do something better, come up with a simple prototype that demonstrates that it's better. Opinions are great, but code talks.

Answer 13

The best way I have found is to relentlessly ask - again and again - 'What is the business problem we are trying to solve' and 'How does this decision help to solve that problem'.

I find that too often folks jump to solutions instead of being crystal clear on what the problem is.

So in your example of how to organize a database, my question would be 'What do we think are the transaction requirements for this project today, next month, next year, five years from now'. It could be that it makes sense to spend a lot of time to get the data model right, it could be a waste of time. You don't know what the parameters are until you get the problem definition clear.