Why is it still so hard to write software?

Question

Writing software, I find, is composed of two parts:

1. the Idea, and
2. the Implementation.

The Idea is about thinking: "I have this problem; how do I solve it?" and further, "how do I solve it elegantly?" The answers to these questions are obtainable by thinking about algorithms and architecture. The ideas come partially through analysis and partially through insight and intuition.

The Idea is usually the easy part. You talk to your friends and co-workers and you nut it out in a meeting or over coffee. It takes an hour or two, plus revisions as you implement and find new problems.

The Implementation phase of software development is so difficult that we joke about it. "Oh," we say, "the rest is a Simple Matter of Code." Because it should be simple, but it never is.

We used to write our code on punch cards, and that was hard: mistakes were very difficult to spot, so we had to spend extra effort making sure every line was perfect. Then we had serial terminals: we could see all our code at once, search through it, organise it hierarchically and create things abstracted from raw machine code.

First we had assemblers, one level up from machine code. Mnemonics freed us from remembering the machine code. Then we had compilers, which freed us from remembering the instructions. We had virtual machines, which let us step away from machine-specific details.

And now we have advanced tools like Eclipse and Xcode that perform analysis on our code to help us write code faster and avoid common pitfalls.

But writing code is still hard. Writing code is about understanding large, complex systems, and tools we have today simply don't go very far to help us with that. When I click "find all references" in Eclipse, I get a list of them at the bottom of the window. I click on one, and I'm torn away from what I was looking at, forced to context switch. Java architecture is usually several levels deep, so I have to switch and switch and switch until I find what I'm really looking for -- by which time I've forgotten where I came from. And I do that all day until I've understood a system. It's taxing mentally, and Eclipse doesn't do much that couldn't be done in 1985 with grep, except eat hundreds of megs of RAM.

Writing code has barely changed since we were staring at amber on black. We have the theoretical groundwork for much more advanced tools, tools that actually work to help us comprehend and extend the complex systems we work with every day.

So why is writing code still so hard?

Answer 1

Code Bubbles goes some way to solving this, and is a step in the right direction. We need to be taking tools like Code Bubbles, clang's static analyzer and call graph visualisations for granted in every language, on every system.

Answer 2

So why is writing code still so hard?

Answer is very simple. Because, most of the time software projects are managed by cluless managers, who have no idea what it takes to write a software.

Answer 3

There is nothing wrong with something being hard, after all that's how you get paid the big bucks. Seriously, though - we are always solving custom problems and trying to fit the best technology available to them. It is hard to predict every single thing that can go wrong, and separation of concerns go a long way addressing that. Interface oriented development, TDD, IoC and many other concepts, when applied properly, if don't make software development simple at least make it manageable.

Answer 4

Writing code is hard. The reason is, IMO, we always try to do something which is not done before. Imaging how long it took a write a nntp newsreader in c 10 years ago, and now we can probably write an iPhone rss reader in days.

Answer 5

Determining all the rules needed to handle every single case in a program is where the headache is to my mind. IMO, most people don't understand that part of writing software is handling all these strange cases where something may go wrong and it has to be handled gracefully, meaning that if someone enters the wrong value for something, the system shouldn't kill itself for allowing that. Building robust systems that can handle human and mechanical error while still having security, auditing and all kinds of other features can be hard for a number of reasons.

Just to give an example where I work, we have spent nearly 2 years working on replacing a CMS because of all the other things it touches like the ESB or CRM for example. Part of the challenge is integrating the system with people and part of it is "Humanity: Epic FAIL" that if you haven't seen that talk by Jon Skeet, you should definitely check it out.

Answer 6

Just a few points/criticisms... if I seem mean I don't mean it; I quite like your question.

The Idea is usually the easy part. You talk to your friends and co-workers and you nut it out in a meeting or over coffee. It takes an hour or two, plus revisions as you implement and find new problems.

No offense, but when it takes "an hour or two" you're already down the path to "hard", "mismanaged" and "failed project". There's a whole lot between those two hours you're talking about - which in my opinion doesn't sound a whole lot like "requirements analysis" - and the implementation phase. The entire "process" you describe lacks consideration of stakeholders and risk management. (that's just my take)

When I click "find all references" in Eclipse, I get a list of them at the bottom of the window.

That sounds like high coupling to me... also indicative of some glazing over the design phase between that coffee and the implementation. ;)

The more important point, however, is that writing code isn't hard. Hell, writing software isn't hard either; what's hard is delivering solutions or equally developing a product someone will use... and hopefully pay for.

This is also why punch cards were hard and had to be left in the dust; writing programs for calculus (something well defined and formal) was hard enough on those things, but moving into the realm of more ambiguous "business needs" would have been impossible.

Or in other words, what was hard was traceability. No one could look at a punch card and explain why it's in an ERP system, for example. As a result, the system would start to lose conceptual integrity and slowly stop delivering on intended users' needs, at which point not only is developing the solution hard, but writing the software becomes hard too.

On the other hand, if I'm working on ( * thinks * ) Warehouse management software, and there's an object called BigBoxFullOfStuff or ScheduledShipment it's a little easier to figure out why that's in the system.

In the end, we can get and develop all the tools and widgets and IDEs we want, but if we still work and develop like code monkey cowboys, writing software will be hard.

Answer 7

Because expectations of features and requirements of software are keeping pace with improvements to development tools. 50 years ago creating software to add a few numbers together was difficult, but expectations were low. To write the same software now can be done in literally seconds, but users want more.

Answer 8

Writing code is hard because it requires concentration.

Answer 9

You can't beat the insight of Edsger Dijkstra's 1972 Turing Award lecture, "The Humble Programmer:"

as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming had become an equally gigantic problem. In this sense the electronic industry has not solved a single problem, it has only created them, it has created the problem of using its products.

To put it in another way: as the power of available machines grew by a factor of more than a thousand, society's ambition to apply these machines grew in proportion, and it was the poor programmer who found his job in this exploded field of tension between ends and means.

and towards the end:

As an aside I would like to insert a warning to those who identify the difficulty of the programming task with the struggle against the inadequacies of our current tools, because they might conclude that, once our tools will be much more adequate, programming will no longer be a problem.

Programming will remain very difficult, because once we have freed ourselves from the circumstantial cumbersomeness, we will find ourselves free to tackle the problems that are now well beyond our programming capacity.

The whole lecture is well worth reading.

Answer 10

Actually I found the complete opposite to be true.

Its not hard to implement once you have all the information. It really isn't. My group just finished a release and an enormous amount of time was spent on the HOW the system should work both UI and backend.

The problem is that you usually do not have all the information a priori. You have to go back and forth with the stakeholders on how the system should work.

Some amazing problems have been solved with programming languages and a very short amount of time. Just google the numerous programming challenges like the ICFP .

I would say good software tools/programming languages allow you to model the problem so that you do find questions that you did not ask about the problem (IMHO very strongly typed languages like Haskell are good at that). However an even more important criteria of programming language/tool is how easy it is to make changes.

Some programming languages are easier to make changes than others and I would gladly trade opinionated-ease-of-use language to very adaptable language. Hence I like languages that make it easy to re-factor and write unit tests.

Besides constant change with Real World Problems/Projects Another is communication. But communication not with human to computer but human to human. The human to human communication component is where I think we need more research.

So to sum it up: I believe change and communication are the real challenge not the problem itself.

Going back the recent release I would say the time break down was as follows: 70% Design 30% programming.

Then of that 30% programming:

• 20% Unit Testing code
• 20% Implementing
• 40% Bug fixing

The bug fixing was in large part because of not enough unit tests and some design problems.

Answer 11

Writing code is hard because solving problems is hard.

And the simple truth is that the problems are rarely all solved until the code is written.

Actually spending time to write the code means coming up with a plan for how to proceed under every possible condition, which places the programmer directly in front of all the unsolved problems. So the programmer finds himself either having to solve these new problems himself, or going back to the design committee with a whole new batch of nit-picky questions, like "What do we do when a customer returns a product after it's shipped but before we receive the tracking number?" or "What do we use for our invoice template when the customer purchases more products than will fit on one page?"

In fact, I am right now tackling an issue that wasn't noticed until I actually got into the code. I'm writing code for tracking recently-used menu items in order to to provide an quick list of the things you use the most... except that some of the items are auto-generated, which means they don't have ID numbers, which means that I'll actually have to merge in a second list which is keyed on.... And then I decided that it was time to take a break and read SO to clear my head.

Answer 12

Other answers make some excellent points, but there are a couple things I've not seen posited that I wish to mention.

First, I believe that the problems we're trying to solve with software are (at least sometimes) more difficult. When I first got into the field in the 1980's, it seemed that we were frequently writing software to handle relatively easy tasks such as handling payroll calculations or keeping track of names, addresses, & telephone numbers of customers, employees, etc. Now, we not only have software to do that, but manage customer relationships and predict trends.

Also, the technology is always changing. For example, when I went to college, structured programming was considered advanced and was not universally accepted. A few years later structured programming became accepted and object oriented development was on the leading edge, but not so widely accepted. Now that's fairly broadly accepted, and things like functional programming are moving into the leading edge. (Note that these concepts have been around a while, but the field hasn't used them much in a broad sense for all that long. Also, most of the "proof" that they're better than another method has been anecdotal, considered obvious, or just asserted - without real scientific studies done.) Other examples of changing technology of a different sort: 25 years ago GUI's were new, networking was almost rarely done except within a single data center, and the World Wide Web simply didn't exist yet. Now, these topics are considered standard knowledge and many practitioners are expected to be at least somewhat skilled in all these areas.

Answer 13

Writing code gets easier every year. But expectations of what can be done go up faster than the difficulty of coding goes down.

Think about each successive generation of software. Follow gopher to lynx to mozaic to modern browsers. Look at how far Photoshop has come. Compare the operating systems of today to qdos. Some of the things that were hard to write a generation ago are now too simplistic to be interesting.

Answer 14

http://en.wikipedia.org/wiki/No_Silver_Bullet