I just listened to a discussion about a project drifting away from the direction it's supposed to go. The problem was "bad software design".
Components are not designed to be re-used; the software is hard to maintain, test, extense, and scales poorly.
What did the programmers do wrong? Where/how to learn better techniques?
Of course, there are the GoF patterns... but that's not all, is it?
No it's not. The next step is to learn when not to use patterns.
I saw to many overengineered pattern infected projects.
Doesn't sound like the programmers did much wrong. It sounds like their lacking in experience, and if they learn the lessons, they may (hopefully) do better next time.
Whoever built the team was wrong in not getting somebody on it with some experience.
The only way to improve is to read, try stuff, learn from mistakes, and get advice from people who have some of this experience already.
If you're looking for some good reading material I'd definitely recommend Code Complete 2
Its not just code related but Project Life Cycle related. I found it invaluable.
The quality of software design will improve with real world industry experience.
No university and college degree is a substitute for working in a real job developing real software.
Books such as Code Complete by Steve McConnell are also invaluable resources.
Try reading Fowler's Patterns of Enterprise application Architecture, or any of the business objects material from Rocky Lhotka.
Both are great resources.
Two things I find most useful to avoid project failures like this:
As you mention, patterns. Get a good book on patterns for the target language. I quite like the Head First Patterns book which is Java focused. Just learning a few of these should improve the overall architecture, or at least give the architects some food for thought.
Get an 'under the hood' book for the key architecture platforms. For example, I would recommend that anyone building a system that uses a non-trivial amount of Oracle database should read Tom Kyte's books on db design. Books like his cover common fundamental architectural errors; the sort you need to be aware of before you start drawing system diagrams.
If a key person is saying that the project is drifting away from its direction, then most probably there's a gap between design/development and basic requirements of the project.
Might be because of bad design as well, for that I'd agree with Eoin about Code Complete 2 and I'd also recommend Code Complete.
But to answer your question specifically, reading ONLY cannot help; it has to be a mixture of reading, working with good people and maybe have a review of your current project done by a senior programmer/designer from a different team.
Moving forward, you can have sessions with that person or a particular group on a regular basis (during tea breaks?) and keep visiting forums which discuss design/development/architecture e.g. theserverside.com is for Java. Not to mention that you have to keep experimenting with some stuff every now and then. Moreover, when you design a system, try to have a POC if it involves a new design or a particularly new technology to be used so that alignment of it can be done with the actual project requirements.
Usage of Patterns can follow after one has basic design concepts, because selection of right pattern in right place is more than a basic item.
I'd recommend the Principles of OOD :
Principles of class design
SRP The Single Responsibility Principle A class should have one, and only one, reason to change.
OCP The Open Closed Principle You should be able to extend a classes behavior, without modifying it.
LSP The Liskov Substitution Principle Derived classes must be substitutable for their base classes.
DIP The Dependency Inversion Principle Depend on abstractions, not on concretions.
ISP The Interface Segregation Principle Make fine grained interfaces that are client specific.
Principles about packages In this context a package is a binary deliverable like a .jar file, or a dll as opposed to a namespace like a java package or a C++ namespace.
REP The Release Reuse Equivalency Principle The granule of reuse is the granule of release.
CCP The Common Closure Principle Classes that change together are packaged together.
CRP The Common Reuse Principle Classes that are used together are packaged together.
The last three principles are about the couplings between packages, and talk about metrics that evaluate the package structure of a system.
ADP The Acyclic Dependencies Principle The dependency graph of packages must have no cycles.
SDP The Stable Dependencies Principle Depend in the direction of stability.
SAP The Stable Abstractions Principle Abstractness increases with stability.
Try reading Eric Evans's Domain-Driven Design: Tackling Complexity in the Heart of Software along with Fowler's Patterns of Enterprise application Architecture
Components are not designed to be re-used; the software is hard to maintain, test, extense, and scales poorly.
It's really expensive to design software that can be reused, at least three times more expensive than the nominal case. My bet is that the project estimates and budget didn't reflect this, in which case it's not a failure by the developers. It's a failure by the project manager to understand the implications of designing and developing for software reuse.
So I think you should ask for much more detail, rather than let some random managers decide why the project drifted.
As for learning good design and architecture, that's hard. I think most of it comes from experience and learning from mistakes that you and others have made. After a few years, you develop a "taste" for what works and what doesn't.
If you use Visual Studio, layer diagrams and layer diagram validation in Visual Studio 2010 Ultimate can help your project stay within the specified design. Layer diagrams describe the logical architecture or design of your system. Layer validation identifies code that conflicts with the design prescribed by the diagram, which you can create from or map to solution artifacts, such as namespaces, classes, methods, projects, and so on. You can include layer validation as part of the build process and gated check-in to catch these conflicts during those operations. VS 2010 Premium supports read-only versions of the layer diagram, but it does supports layer validation.
VS 2010 Ultimate also includes other useful tools to help you see where the code might have gone astray. You can create dependency graphs from .NET application projects and assemblies. These show you existing relationships in your code and where there might be undesirable patterns or dependencies. You can generate sequence diagrams from .NET code to see how the code implements a particular method. The current release supports only .NET code, but the team is working to provide additional support in future releases.
You can also help ensure the project meets its requirements by using UML diagrams to visualize and communicate the users' requirements and the intended design. The team is working on additional features to enhance the functionality of these diagrams in future releases. The elements on the diagrams can be linked to Team Foundation work items to help the necessary development and test work and monitor its progress.
The following topics describe this in more detail. I've posted more links on my profile for more info.
How to: Create Layer Diagrams from Artifacts: http://msdn.microsoft.com/en-us/library/dd465141%28VS.100%29.aspx
How to: Validate Code Against Layer Diagrams: http://msdn.microsoft.com/en-us/library/dd409395%28VS.100%29.aspx
How to: Generate Graph Documents from Code: http://msdn.microsoft.com/en-us/library/dd409453%28VS.100%29.aspx#SeeSpecificSource
How to: Find Code Using Architecture Explorer: http://msdn.microsoft.com/en-us/library/dd409431%28VS.100%29.aspx
How to: Explore Code with Sequence Diagrams: http://msdnstage.redmond.corp.microsoft.com/en-us/library/ee317485%28VS.100%29.aspx
Using Models within the Development Process: http://msdn.microsoft.com/en-us/library/dd409423%28VS.100%29.aspx
From the information you supplied, I believe the issues are not so much software design and architecture related but more project management related.
I recommend:
The Personal Software Process helps you improve your development cycle. One good tool is the software review check list. The course also extends into the Team Software Process, which adds team interactions and scheduling concepts.
The next concepts I apply are Test Driven Development (TDD), and the Spiral Development Model. I split a project into phases. The beginning phase is to implement a skeleton, foundation or framework. The next phase is minimal functionality or a strawman for demonstration purposes. Additional features are added in phases. This follows nicely in the Spiral Development model and Test Driven development, allowing easy progress tracking by management and sharper focus by engineering.
During each development phase, software can be refactored. Through software evolution (my term), with refactoring, working pieces of code are modified less and less. A stable foundation is built. TDD brings up the reliability and a armament of test suites.
Finally, Design Patterns allows you to avoid thrashing about in issues that have already been solved. The book Code Complete will help you avoid wasteful practices. These concepts have helped me produce quality software in record fast times.
-- Thomas Matthews, programming the world one bit at a time.
NDepend documentation comes with some pretty cool and advanced online blog posts, articles and white books concerning architecture of .NET code:
Advices on partitioning code through .NET assemblies
Control Components Dependencies to gain Clean Architecture
Re-factoring, Re-Structuring and the cost of Levelizing
Evolutionary Design and Acyclic componentization