How do I build something when I know I'll get it wrong?

Question

Background

I have a personal project that I've been trying to build for around 5 years. In essence it's an online game - a web application. It's not a "money maker", just something that I really want to build, so finding the funding to hire a skilled team is very unlikely.

I've built two fully-functional prototypes over the years, both successful from a concept/user testing perspective, but both spectacular failures from an architectural perspective; the code was a mess, couldn't be maintained or further developed, and had to be thrown out.

It took a good couple of years to acquire the skills required to build the client - which is rich/stateful and pretty complex. I aligned my career and studies towards this side of the development divide. I'm finally at a point where I can build a decently architected, sophisticated client that can grow and doesn't need to be thrown out 6 months down the line. There's a ton of work to do on that front, but at least I know I can do it, and do it reasonably well. The back-end is another story.

So far I've rebuilt the back-end at least 11 times with various combinations of PHP, SQL, Ruby, CouchDB, MongoDB, FriendlyORM, NodeJS etc etc. I don't usually get very far before I discover some huge flaw with my approach and start over: RPC to REST, relational to document driven. I'm well aware that premature optimization is the root of all evil, but the application is very dependent on fast moving, highly dynamic data. RESTful API design, scaling, sharding, caching, auth, replication - I don't have much experience with any of this, and I can't expect to be remotely decent any time soon. These things requires years of study and experience.

It makes more sense to find an expert in this field, but without funding I feel that I need to successfully deploy another prototype in order to attract the right person. So, I shall just have to build it as best I can.

The Question

Assuming that however I build it, the back-end architecture is going to be wrong and will need to be rebuilt, what's the best way to proceed with building "just enough" to continue development of the client application? Even if it's nasty, is there a way to "throw together" a JSON web service? Ruby with Sinatra and MongoDB? Django? Is there some out-o-the-box webservice builder? There's no need for a full-stack web framework as there is no presentation layer - just data. Any advice would be greatly appreciated.

Answer 1

You don't need to build any kind of web back end to be able to get on with prototyping the client application. Just make the client application call stub functions that return dummy data.

Answer 2

My opinion: Too much emphasis on the technology and not enough on sitting-down and doing the proper designs.

1. Start with a high level design.
2. Identify the different, major pieces involved. Spend some quality time for step #1 & 2.
3. See what off-the-shelf components you can use to help implement the different pieces rapidly. Consider that you may later rip-out these components for something else (including your own solution).
4. Revisit #1 & #2
5. Pick a piece or two and start coding a working prototype for the pieces involved.
6. After having done the legwork, start again from step #1 and see what has changed so that you can compensate accordingly.

Answer 3

Make it work slow, first, with clean, modular code.

If it's modular, you can replace a layer or two without having to scrap the whole thing.

While they provide modularity, be careful of web services, even REST, as they tend to be slow; there's a lot of overhead with each connection, for example.

Answer 4

Building a large, complicated application of this sort, especially one with lots of interdependencies, state-specific conditions, and client-server divisions that may require the use of incompatible languages, is daunting no matter how you approach it. Based on my experience with other projects of this sort, you're destined to fail on your first try no matter how careful you are. The trick is to embrace failure as an inevitable step along the way to success and not fuss over every little thing as you build out the application.

The first mission should be to get it "working" with as little programming as possible, to simply get the effect you're looking for, even if very roughly, so you can see how it all fits together. If you can break down the big problem into a series of smaller problems to solve, you may find success with one element and that can be motivating to tackle bigger or different problems.

A useful strategy to employs is to keep the elements of the application loosely coupled, to avoid interdependency except where strictly required, so you can swap out or make improvements to portions of the whole without having a cascade of consequential changes. For instance, your networking code could be capable of transmitting state changes between client and server without caring about the nature of the states themselves, but your state management code would not have to care how the states are transmitted, only that they will be.

It's also useful to have a handle on the overall architecture of the application so you don't get lost in the weeds. From a high-level perspective, you may want to be familiar with basic Design Patterns that can help you organize an otherwise impenetrable mess of code into something simple, modular, and easy to build out.

On the subject of frameworks and languages, I'd say avoid switching so often. While it is educational to explore a new language to see what features may help with your particular problem, you will probably be more productive if you stick to one, even if it can be difficult to achieve some things, because you become more effective with it, improving your approach to better suit the language. While Haskell might be a better fit for some problems, even plain old PHP can be coached into doing the exactly same things with enough determination.

There is a temptation to try new things, to broaden the scope of the work to have it "done right", to build in new functionality as it occurs to you, but to keep the project under control you'll have to maintain the discipline to avoid these expensive and distracting activities which are often, taken objectively, only flights of fancy or premature given the overall state of the project.

To specifically answer your question, build it in the framework you're most familiar with, where your strengths are, and do it in smaller increments that produce useful results. Maybe that's the client display engine, or the networking component, or the back-end state transitions, but whatever it is, you should have it in a "good enough" state to start attaching other components to it.

Solving ten little problems can be tedious and time-consuming, but it's a lot easier than solving one gigantic one.

Answer 5

Johnny G pretty much nailed it with his comment to your original question. The situation you describe even happens to fortune 500's believe it or not. You need to thouroughly plan what it is you are trying to build/accomplish with your project before choosing and throwing out the newest and coolest technologies every three months.

I think this article from wired, "learn to let go" about the failure of duke nukem forever to end up getting shipped, will explain this better than I can.

http://www.wired.com/magazine/2009/12/fail_duke_nukem/

(it's also a pretty fun/informative read regardless)

Answer 6

If your project sucks and no one ever uses it, what will it matter how optimised it is? Get an end to end version working, I'm sure you will discover a lot of other issues you haven't considered yet, which are likely to be of greater importance.

Answer 7

I find there's only one way to get a personal project to completion.

Code smart first, plan later. Develop that prototype, but design it so that any single piece can be removed and replaced by another piece.

If your language-choice is ruby, for example, build your classes to have a well-defined interface that you will never break. Worry about each function "doing" the right thing, not really caring how it does.

Then, go back to your prototype built modularly, and fix one piece at a time.