Overengineering - How to avoid it.

Question

Sometimes I find myself designing my classes for a certain project. I start with some entities, some interfaces, but after some time I think:

Hey what about creating a handler for the classes using a Factory Method, Strategy, Using Generics, etc, etc, etc.

At some point when I look to my classes I have a lot of generics, small objects, everything turn so complicated that I feel desmotivated. I turn a simple project into a NOT COMPLETED PSEUDO FRAMEWORK. And I,ve never created a framework before.

How do you battle the impulse to complicate things? How do yo recognise the point where you should stop the utopic design and start doing working things?

Any advise? How you do it?

Answer 1

To start with, use the XP rule: "Do the simplest thing that can possibly work."

As things get bigger, you have to ask some other questions.

• "What is the real workload going to be?" -- that is, what is really going to be done with the systeem, and when

• "what are the most probable uses of the system?" -- don't build lots of complexity for improbable edge cases

• "What are the risks." Short version of a war story: I was preaching simplicity to a bunch of colleagues who had the over-engineering bug. I wasn't successful. They proposed a system that was 10 times more expensive than the customer was really thinking. Why? because they had turned a bank's idea of a high reliability system into a real-time life-critical system. 0.99995 availability where the bank needed 0.95 -- and sure enough, cost is exponential in the number of 9's.

• "where are the requirements most likely to change?" This is Parnas's Law: modules should hide potential requirements changes. But you have to think about which ones are probable.

• refactor for simplicity

• there are people who feel "why make it simple and efficient when it can be complex and wonderful." Kill them. Think of it as evolution in action. Or if you can't kill then (people are so stodgy about that nowadays) at least keep an eye on them, and if they can't be trained out of it, move them someplace they will be better suited. Tax law seems to be a good spot for them.

Update

Let me add one more rule. "Don't give yourself much time." If you use short iterations with working code at the end, then you are naturally prevented from getting too cute. Charlie's Law of Software is that all successful projects have working code within 90 days of the project kickoff party.

Answer 2

I tend to rely on Use Cases: put yourself the end-user hat and see how the system should work.

From that, take some entities and how they would behave in the real world - there you have your basic API.

From there on, I usually go ahead and code whatever is necessary to make it work. You can always change it later - if the project becomes larger, then you'll refactor as necessary, but getting your hairs out trying to make the perfect framework / project from scratch is usually a bad idea.

Remember good frameworks are not achieved in just one iteration; they are built and refactored [too] many times, until the desired features are there.

Answer 3

Analyze what you've done in the past, and see where you've actually repeated code or actually had a reason to make something generic. That's a good starting point. Remember, the reason there isn't one unified framework for all things is because as programmers and developers of products, the patterns aren't always the same. As a developer, you might find yourself doing a lot of the same types of things for a number of reasons...just your personal tendencies or maybe you work within a given industry or target a particular platform, so your framework should be custom to your own needs. To me, this is only exposed over time.

I personally start off with a framework, but I only add things to it as I feel they need to be added. Over engineering can not only waste time, but it can get you into huge trouble later. If you spend too much energy making things generic, they can become completely incomprehensible to anyone other than you. I've seen it happen. Not only is it hard for another developer to pick up your work or collaborate with you, but you'll find yourself looking at something you spent a ton of time making generic only to realize it would probably be cleaner and more maintainable in the future if you didn't.

Answer 4

The first thing you need to keep in mind is YAGNI. You Ain't Gonna Need It. Until a certain feature, principle, or guideline becomes useful and relevant, don't use it.

Until you have more than, say, three instances of classes that look like they're doing the same thing such that you can abstract them to a generic class, don't introduce generics. Until you have a second class that you might need to implement the same methods in, or until you need to decouple this assembly from another assembly, use concrete classes instead of interfaces. Don't use this nice-geeky design pattern (did I hear Factory?) until your classes start looking like they could use a Factory.

Complexity in the manner that you talk about often arises from perceived needs that aren't realized: don't attend to these needs unless they become real.

Answer 5

I think the key is to adopt the mindset that you simply can't plan for every contingency. I believe that you should only plan for foreseeable change and code reuse. If you try to make your design infinitely flexible and reusable, you'll likely draw all your lines of abstraction in places where they just get in the way instead of helping. The rule of 3 is a good rule of thumb: It takes at least three examples of how something might be used to figure out the proper way to abstract and generalize it. My attitude is, if I can't see specifically how I could want to extend or reuse something, I design for readability or simplicity of implementation instead.

Of course, when your clients expect stable interfaces and you're stuck with any underengineering you do, this complicates things. However, at this point, it becomes more of a political issue.

Answer 6

I like the old Einstein quote: Make it as simple as possible, but no simpler. Sometimes I will try to oversimplify the solution and that can be just as bad as over complicating it.

Answer 7

What I do in the "fog of war" is: try to hack a simple solution by making little "experimental" pieces of code for what I think are the most complicated parts of the feature/system. Then, extract from the "hackish" code I created the pieces and knowledge needed to write a simple but elegant version.

Answer 8

Write a unit test. Do the minimum amount of work to get it to pass. Commit your code. Repeat ad nauseam.

Answer 9

This is sometimes called "Analysis paralysis".

Here you can find description and some possible solutions.

Answer 10

Consider the real-life designs that did scale well (TCP/IP protocol being a prime example). Their engineering doesn't involve thousands of abstract objects or any of that "crap".

Another example: the linux kernel is written in C (not an OOP language at all!)

Maybe sometimes it's better to sit down on a clean desk with a pen and paper and think properly about a simple design.

Think in terms of the big picture; the major components. Don't turn every little minute detail into a full blown class. Also, "just do it!". Sometimes, you have to force yourself to be a "bad guy" and just write the dam code without abstractions.

You can always refactor later.

Answer 11

Write code that implements clean, straightforward, simple objects where each object does no more than necessary, even if you "might need it later." Code that is well written and that has good unit tests will be easier to refactor later when you need to make changes.

In my experience, people often badly predict what will be needed "a year from now" or later, putting too much design into some areas, too little into others.

Focus on what you need right now. Implement that cleanly, with good unit tests. Then you can always refactor later. When you know that you can always refactor later, then you'll feel less tempted to do it "perfectly" the first time.

Answer 12

my first line of defense to over engineering solutions is to talk with a knowledgeable friend or colleague, often they will point out if something is overkill.

Answer 13

If this 'overmodelling' stops you from getting work done, you could try to make a 'quick and dirty' but working (to at least some extent) version (for the crucial parts) and then start refactoring / remodelling from there on.

I realize this is harder in some languages than it is in others. Most modern (and especially dynamic) languages make quick refactoring easy.

Answer 14

My way of avoiding the risk of overengineering things is the way I learn things. When I read stuff (e.g. a book about design patterns), I do not try to memorize it all. (Unless I'm scheduled for an interview) I just keep in mind what kinds of problems can be solved with those patterns. Until I encounter such a problem, the solution is simply not on my mind.

Answer 15

Thank for the responses. So far to avoid overengineering we play with concepts:

"Yagni" "Do the simplest thing that can possibly work." "quick and dirty' but working" "old Einstein quote: Make it as simple as possible, but no simpler."

and so forth.

The problem is within my analysis capacity and my unwillingness to do simple things.

I will review the post to cure my APETITE for COMPLICATED THINGS.

Best Regards!

Answer 16

Good rule of thumb:

The first time some work comes in, just do it.

The second time similar work comes in, maybe just do it.

The third time similar work comes in, start abstracting and automating--you'll be getting more of this kind of work.

Answer 17

Perhaps trying to utilize TDD (test driven development) - first you write the test, and then implement the solution. It doesn't work in every situation, but it certainly does in some.

Answer 18

I suppose that the thing that best keeps me from going overboard is to remember that anything I want to add now, I can always add later. The code is just sitting there waiting to be overengineered whenever I feel like it. :-)

However, this relies on having a good enough and fast enough testing regime that you can afford to rewrite something just for the hell of it.

Answer 19

From linux kernel modules documentation - "If you don't know what it is say no"

In programming it means to me - "If you don't know what it is remove it" :-)

Answer 20

OPINION: well, overengineering is a problem that you'll get when using paradigm that is strongest in inventing and adding new stuff. OOD is focusing on adding stuff to the system, not taking it away. It helps you to start going, provides you with simple abtractions, as long as your are starting from scratch. Then the amount of high-level abstractions just keep on growing in number (coming closer and closer to complex low-level problems), and it becomes impossible to upkeep all the dependencies and innnerconnections. So, we turn to refactoring and hope it's not too late to turn the titanic around. But IMO issue of growing complexity should be addressed right from the start. It should be build-in into the programming paradigm itself.

Answer 21

I've noticed I suffer from this problem as well.

The first projects I ever did, I felt I had to go back and fix the messy code whenever I could. Thought about it all the time. They were crazy to maintain. Then later on, in later projects, things would go a bit better, prettier code. I didn't feel the same. Then in other projects (these are all small C# desktop apps mainly) things would go better and better.

I think that as programmers we need to keep coding and coding to really understand when we need to stop, or start. I'm still a victim of over-engineering though, and there's also something saying "One day a programmer will look a this code.. what are they going to think? How are the going to see it?".. and every programmer will think "Damn! I have to impress.".

This can also lead to that. I've broken software and made it too complex sometimes for the size and capability it was meant for.

Many, many times I've created classes that soon after were no longer needed, and then eventually discarded them. A lot of time goes wasted, a damn lot.

I now believe thanks to these posts in the time frame problem: if we have too much time, we'll waste it. Our brains as developers must have a funny neurology, since we start to 'refactor' our lives sometimes.. thinking of data structures before we sleep.

I've asked my boss to have a programmer working with me on day, someone that has enough guts to sit there, go line by line, and one that's able to criticize. But beware, not too much. Pair programming can help, and discussing the architecture with someone that has done more OO and less code in their lives could introduce some new views. Also pick up 'Object Oriented Software Construction', by Bertrand Meyer.

All the best and happy engineering.

Answer 22

YAGNI. KISS. As others already pointed out...

In practice when I work on designs (on paper, not on computer, not on code), I regularly estimate the time the implementation would take in man hours, days, weeks and so on. This estimation may not always be correct, but if its done regularly you get a feeling for it!

As a last consequence I negotiate work vs. spare time.

But this IMHO only works when you separate designing and coding. Designing on code is much more time intensive and the designs tend to follow practice with existing code, which usually leads to bad compromises... and very often to overengineering: you will be driven into situations e.g. where you have to make some part A compatible with another part B and will start throwing patterns at it to solve the design problem. But if you design on paper (ideally without any existing code), then you still concentrate your design efforts on the real problems that your software should solve.

However, designing on code is not always avoidable.