When developing software I usually find myself continually questioning "Is this the best way?", "Is there better technology for the job?" and as a result I put more effort into actually investigating & researching different design patterns/technologies/best practises than actually just developing the thing with tools I know will work!

A common thing I tend to do is think too far ahead and worry about things that may happen rather than focussing on things that will happen. I don't believe this is necessarily a bad thing, however, I find it sometimes takes up a little too much of my time.

I would like to hear if anyone else has similar issues and how they go about tackling them?

+7  A: 


To prevent over-engineering, do this.

  1. Build the smallest, simplest thing that solves the problem.

  2. Investigate alternatives.

I don't really agree with either. It takes years of experience and a lot of thought to come up with the smallest, simplest thing that solves the problem. Also, investigate alternatives easily becomes a game of its own (i.e. you never stop). So to get some points for your answer, please give some pointers how to achieve these goals.
Aaron Digulla
The smallest, simplest things requires the least work on your part. Investigating alternatives (after you've shipped something) can be a hobby or pass-time. You already shipped something. Once it's shipped, you'll NEVER finish supporting it, so investigate all you want. You'll never finish support, either.
Investigating alternatives is something I have seen far to little of among developers. Sometimes the third or fourth solution considered will shave months off of a project, even if you're already in the middle of working on the first solution. But this takes creativity and effort, and it is difficult to do. Mental inertia should never be allowed to wreck a project.
Jeffrey L Whitledge
@Jeffrey L Whitledge: There's a fine line between "finding a better solution" and "spinning wheels uselessly." In this case, someone is asking for help in how to reduce the wheel spinning. In other cases, you may be right -- not enough engineering is being done. However, I've worked with people who prefer to study the problem rather than solve it. I tell them to just build something; they can study all they want -- after they've built something that works.
+1 This approach I like. Get the thing working `then` worry about making it better :)
@S.Lott - I am not talking about endlessly analyzing the problem. I am talking about scrapping semi-functional code that requires endless tweaking in favor of a whole new ground-up solution that can invisioned with the experience that came from the first attempt. The solution may even be "obvious" once the idea of examining alternatives is proposed. Programmers are often fearful of scrapping code that kinda works. My experience has been that such leaps can pay off in a big way once the decision is made to try it.
Jeffrey L Whitledge
@S. Lott, I'd modify your two steps to something like:0. Give the job to someone who's already too busy.1. Watch them build the smallest, simplest thing that solves the problem.Profit!
Bob Cross
+44  A: 


Russell Newquist
+1 There's a psychology study somewhere that confirms this. I'll leave it as an exercise to the reader to find the research.
Stefan Kendall
So elegantly laconic
Tim Post
I'd research it, but I have another deadline.
Jarrett Meyer
I like the whooshing sound they make as they fly by.
@Rorschach, "I love deadlines. I like the whooshing sound they make as they fly by." Douglas Adams
Bob Cross
tinkertim, I would've made it even moreso except that my original answer, merely "Deadlines." didn't meet the minimum character limit.
Russell Newquist
Fixed it for you.
Michael Myers
Awesomesauce. :D
Russell Newquist
+22  A: 

Adopt an XP idea...YAGNI. (You Ain't Gonna Need It)

Code enough to solve what needs to be done now. Get it working to your customer's satisfaction, that's all that really matters anyway. Then move on to the next feature.

If you find you need to refactor something, so be it, just do it incrementally.

Ewan Todd
+1 I would say this is probably the best approach to take.

As long you meet your time goal - invest as much as you can. If you can't meet your time goal... invest only if you think it is crucial to meet requirements or if you think you're going a direction that will be impossible to fix later if wrong...


Premature optimizations and handling of what ifs can definitely be a problem.

One general thought is to just make sure that you have code that does the best you can and be willing to adopt better practices as you learn them.

In general though, the simplest answer to solve the current known problems is generally the best. Your code however should be able to catch the unexpected error cases and log/expose them for you to add more robust handling of those corner cases.

Bryan McLemore
+13  A: 

Yes, I've seen and experienced this problem many times. The number one solution (for me) is a schedule. Not a marketing department schedule that gets determined by insert Black Magic here. I'm talking about something like a monthly / quarterly schedule that you inflict on yourself or your group where you say "on this date, we must have a buildable, working project that, if they cancelled our project that day, they'd still have something good."

What this does is put real milestones out there that you have to commit to. Remember, milestones are rocks. They don't move. Likewise, the calendar just ticks by. If you can't commit to coming up with a good-enough solution in time, you won't have anything worthwhile on that day.

Personally, I think a three weeks of development + 1 week of integration, testing, clean-up and final prep is a nice arrangement for small to mid-sized groups. Your mileage will vary.

Bob Cross
Wish I could do plus a googleplex on this answer. We now have shorter milestones, and it has clearly improved our morale + output.
Another thought on this is: if you need to redesign it (ie, your original design is bad and doesn't meet requirements well), it's not that hard, you can do it in (an)other 4 week segment(s).
@sheepsimulator, right - when I was making the original business case for this monthly process, I specifically said that this builds in the opportunities for customer change of course. Every customer will eventually say "that isn't what I want!" whether or not that's what they originally said. Worst case: we went the wrong direction for 30 days.
Bob Cross

I run into this often. If I need to solve something in a language I regularly use, which today is JavaScript, and I'm stuck, I try to solve the problem using a new framework. There's something about using a new tool, a new code library, even a browser I don't usually use, that helps me get past the psychological block of trying to do it right.

+4  A: 

You may have to read The Duct Tape Programmer.

Practice YAGNI (You aren't gonna need it), and your productivity may rise a lot.

+1 The Duct Tape Programmer
Ewan Todd
-1 The Duct Tape Programmer
Arnis L.
+1 The Duct Tape Programmer
-1 The Duct Tape Programmer-- sounds too much like a project manager fantasizing.+1 "Worse is Better"-- a related but more practicable notion.
+1 YAGNI, -1 Duct Tap Programmer.
Jeff Sternal
+2  A: 

This is one of the things that a review with your peers should help determine. They should let you know if 'you going into the weeds' (and be able to justify that assertion). On the flip side, they should also let you know if you haven't done enough and are designing something that brittle and not resilient enough to change or problems.

Michael Burr
This is critical. However it requires the person who brings the design to have the humility to accept input from fellow employees.
Robert Gowland

The idea of incremental deliverables focussing on the most critical features from the Unified Process was designed to solve this problem. It takes discipline to implement.

Make a list of features (use cases), prioritize them, choose the highest priority and work on that as if its the only feature you'll ever need. When you deliver it in working form, analyze again and select the next feature set to work on.

Vincent Ramdhanie
+5  A: 

In my experience, the most simple way to avoid over-engineering is experience. While greenhorns will struggle endlessly with their doubts and fears, a senior developer will just do it. And they will get it right (the second time, if not the first).

The second most simple way is confidence. Here lies a danger, though: Without experience, confidence can be extremely dangerous.

Which leaves the question how to arrive with both in a short time. The world's greatest minds are working on it. My path is using automated tests. That kills my fears and doubts faster than anything else. It allows me to move knowledge and experience from my brain to the computer, so my poor little brain is free for the next thing that comes up and I don't have to worry about all the things that I've already solved.

Aaron Digulla
+1 would have to agree with this. I sometimes find myself doubting my own initial approach simply because there is someone more experienced telling you that X way is better.
+3  A: 

Hire a bunch of software architects to form a committee to analysis all designs.

Designs will be submitted in UML to ease the analysis.

All projects will all use an in-house XML based language to avoid the specifics of a single language.

You will also need a detailed coding standards to avoid the workers over engineering things. This should only cover important things like the positioning of { }

Martin Beckett
I once heard of a tongue-in-cheek project to create an application to print the verses of "The Twelve Days of Christmas", following SOPs, complete with 3 levels of specs, staffing plan, budget, schedule, etc.
Mike Dunlavey
... and you're right. Musn't forget "coding standards"!
Mike Dunlavey
  • Choose the solution that works for you and don't try to find something else unless you get limited by the current one (Murphy: If it ain't broken, don't fix it)
  • Create a list o priorities and stick to it.
Victor Hurdugaci
+1  A: 

Use part of CMMI; measure yourself.

Figure out a way to keep a running tally of how many times you overengineer vs how many times you underengineer. Figure out how much pain underengineering something costs you on average. Wait a year or two, and look back at it.

Thinking more about it may help you in the short run, and in the long run, you'll have the data to know whether or not your fear of overengineering was justified.

Dean J
This would be a much better comment without "CMMI". Measurement is not a bad idea at all. CMMI contains many dubious practices.
Sean McMillan
+18  A: 
  1. make it work
  2. make it better
  3. run out of time

In that order.

Kelly French
+1 Another good approach, either way in the end you are going to have a product that solves the problem.
+3  A: 

Release Early, Release Often

Even if you're not releasing to an external client you can still have an internal releases to a product owner or testers. This kind of work cycle makes you focus more on the task at hand and not the future.

+1 agreed, it breaks down the development aswell so it doesn't feel you have a massive workload all at once.


Another approach is to examine periodically all you have done and refactor it to the bones: Remove all that is not needed. Repeat as often as needed. Makes the code leaner for the next iteration.



I avoid over- or under-engineering by trying to balance the amount of time I spend investigating and designing with what I can reasonably expect its use and lifetime to be.

Let's say I'm writing a utility that only I will ever use, and I will use it rarely or even just once. If I have a choice between coding up such a utility in Bourne shell or Perl in ten minutes that takes overnight to run, and taking three hours to write an optimized version using sophisticated and difficult algorithms in C++ that runs in one minute... electricity is cheaper than my time.

On the other hand, I've written key components of products that have been used by or affected millions of people over many years. In such instances, it's been worth it to take the time and effort to put a lot of effort into investigation, research, and design, use the absolutely best tools and techniques, and then polish the resulting code to a glossy shine.

There's no set-in-stone way to do this - it's all judgment. And as with all matters of judgment, experience is very helpful, as Aaron Digulla aptly pointed out.

When I started out as a professional software developer twenty-four years ago, I didn't know jack about how to make these decisions. I'd just write code, and if it was too slow or too buggy or something, I'd go back and fix it. But when I did that fix, I'd try to think about how I could have avoided the problem in the first place, and how I could apply that in the future. And I also have tried to listen to other programmers when they talk about problems they ran into and how they fixed them.

Now, many dozens of projects and perhaps millions of lines of code later, there are a lot of design decisions I can make almost instinctively. For instance: "If you're working in C++, and you're facing a problem that some STL template will solve, and you're not constrained to avoid using STL, then that's the way to go. That's because modern STL implementations are highly optimized, and any improvement you could get from writing your own code would just not be worth the effort."

Also, I can just look at a situation and say, "The 10% of the project where we're likely to have problems is here, and here, and here, so that's where we need to concentrate our research and design effort. And the other 90%, let's just make it work however we can do it." And it works out pretty well.

So keep coding, and keep improving your code, and keep learning from other software developers and from your own experience. If you continue paying attention and thinking about things, increasing software design mastery will come over time.

Bob Murphy
+1  A: 

There are a few ways that come to mind:

  • Focus on what was asked and where possible make the requirements as clear as they can be. The former may be seen as passive-aggressive to some as doing exactly what was asked isn't necessarily a common practice.
  • Stay in the moment, avoid playing the "What if" game, YAGNI.
  • Time-box the work so that it isn't a black hole sucking up all the time available.

A couple of other things to notice:

  • Avoid beating yourself up for doing the job you did do. The past is going to stay the past regardless of what you do.
  • The perfect code for an application is a myth. The enemy of a good plan is the dream of the perfect one.
  • While some analysis is good, use moderation to prevent "analysis paralysis"
JB King

Time boxing is what we do.

We have a 3 day look into an upcoming problem, and try some things out and pick a solution.

After that run with it and deliver early and often, after you've shown some progress then you can refactor if a better answer presents itself.

+5  A: 

Excellent question. Here's what I've found, and it's not all easy to do:

  1. Do prototypes. This way, I get a deeper understanding of the problem than I could ever get by just thinking about it ahead of time, and I never get myself wedded to suboptimal code.

  2. Get experience with actual performance tuning of real software because, at least in my experience, over-engineering of software results in massive performance problems, as in this case. This teaches you what typical design approaches lead simultaneously to complexity and slowness, so you can avoid them. One example is over-emphasis on the intricacies of classes, data structure, and event-driven style.
    (This is the opposite of premature optimization, in which by trying to solve problems that do not exist, you end up creating problems.)

  3. At times, I have taken a really uncompromising view of software simplicity, and it has had the cost of being very strange, it seems, to everyone but me. For example, I stumbled upon the technique of Differential Execution, which shortens UI code by an order of magnitude and makes it very easy to modify but at the same time creates a learning curve that few have climbed.

Mike Dunlavey
+2  A: 

Test-driven development and refactoring together mean you don't need to have the best way up front, or even see how all the details fit together... it's about emergent design.

Reading about the ideas behind this might help you worry less about perfectionism:

Here's how I learned to do it:

  • Learn to write the test first for some simple piece of functionality.
  • Then write the code that satisfies the test.
  • Look for elements of repetition or things you can simplify. BEFORE you try to simplify/unduplicate the code, ensure ALL the test pass so you have a baseline for refactoring.
  • After refactoring run all the tests again - to ensure you haven't changed any behaviours that you care about.
  • Check it in.
  • Repeat... until done...

  • This is easier to do when you're paring with somebody else... preferably somebody who already knows how to do XP.

  • After a while you learn to trust yourself and accept that you don't need to control all the bits of the solution in tiny detail up-front... although it pays to start working on the risky/unknown parts first...

In other words, learn XP ;-)

TDD is not really something I have used tbh, I have had some light exposure to it and I must say it wasn't really something I enjoyed. I found it quite tedious at times.
I think it might depend on the person you're pairing with... I certainly find it more difficult to TDD on my own than with somebody else that thinks like me...
@cartoonfox That link says I don't have permission to access it.
+1  A: 

You need two things: Timeboxing and Peer Review

Timeboxing is as simple as saying - I will spend N hours researching technology to make this work better.

Peer Review means you discuss the problem with other interested engineers.

It sounds as if you are working on your own, which makes Peer Review difficult. I work in a Scrum shop. Part of the process requires that we discuss all fixes and features, then get buyoff (agreement) from the other engineers before writing a line of code. This works out to be the same as 'Measure Twice, Cut Once.' We spend about half our time researching and planning, and it is worth the investment.

Fred Haslam

As a programmer in a successful enterprise, your work is to achieve the-code-that-works state & ship, as an R&D engineer you're free do whatever!

+1  A: 

Keep it simple, stupid

A maxim often invoked when discussing design to fend off creeping featurism and control complexity of development


The best way that I've found to prevent over-engineering is to make sure that you only write code for as much of the specification as you currently know. If you have a class that needs to call one web service and retrieve one type of data don't bother writing some incredibly robust system that can handle every possible case.

That said, this technique really REALLY requires that you create clean, well-written, easily understandable code, and requires that you utilize accessors everywhere. If you don't you'll end up with a refactoring nightmare.

When you write code to satisfy only what you know you'll need at the moment you end up building functionality quickly, when the requirements change you can go back and refactor your code to add the missing functionality. Each time you add a new feature your code will (read: should) get better and better.

There are certainly some overarching design decisions that need to be made at the start of the project, but those decisions are very high level and shouldn't affect your ability to change as the requirements do.

While using this technique you'll want to take a moment before writing any new module to ask yourself if it is needed for the current implementation, if not, leave yourself a comment and write only what is needed.

It seems that what I practice has been formalized as YAGNI(You ain't gonna need it).

There is one sure principle to follow: first get things done, then be smart. You must achieve the first goal, the second you don't.

Nikola Stjelja

It sounds like you don't have a project manager.

You need to steal techniques from the famous Rubber Duck Debugging and apply them to project managment. Pretend that the Rubber Duck is your project manager representing the primary customer, and explain to it that you want to take X hours researching a new technology or new architecture. Now pretend that the Rubber Duck asks you if you think that the new features would be worth X*Y of the customer's money, where Y is your hourly salary plus the cost of your desk and benefits. Next the Rubber Duck asks you if you think that the new feature is worth delaying the delivery of the product X hours.

Answer both of the Duck's questions honestly and proceed with development based on your response.

Incidently, you should probably ask the Duck if he minds all the time you spend on Stack Overflow.