Why does it seem that most programmers tend to write all their code at the lowest possible level of abstraction?

Question

In my thirty years of programming experience, it seems to me that the vast majority of the source code that I have read, and the vast majority of the programmers that I have encountered, tend to write all their code at the lowest possible level of abstraction.

Allow me please to provide an example. I answered a post earlier today in which someone wrote (I am not trying to pick on the author):

SomeObject o = LoadSomeObject();
if(null == o
   ||
   null == o.ID || null == o.Title
   ||
   0 == o.ID.Length || 0 == o.Title.Length
)

As part of my answer, I suggested refactoring this by writing an isPresent(String) method that would combine the null test and the length test into a reusable function. This seems to me an obvious step in applying basic modularity. This method is one of the first that I tend to write as I encounter each new platform and language, since Strings and these tests are essentially universal.

While this example is very trivial, and it was intentionally simplified for the context of the writer's question, nevertheless it reminded me that I commonly see low-level code like this in production use. Of course, my suggested refactoring is really only relevant in a production context as well.

My point is that I don't just see this occasionally. Instead, in most of the production applications that I have seen, all the code is low-level and there are no custom abstractions present at all. Indeed, the code seems to be written as though it is not possible to create custom abstractions in that language/platform/environment.

More specifically, I routinely see much more complicated examples such as database access code written only in terms of language primitives and the bare API calls/types (create a connection, set its properties, open it, construct a SQL statement, issue it to the connection to get the results, walk the results, etc.), repeated in every method that needs database access. Likewise for every programming task that I can think of: UI code, file access code, string manipulation, bit manipulation, etc.

Applying modularity and other such principles tends to raise the level of abstraction, so that I can write new methods at a very high abstraction level based on invoking other abstractions already written. This can eventually coalesce as what some might call a "domain-specific language", in which the entry point (main()) and much of the guts of the final program may read mostly like prose, with almost no low-level constructs present above some sort of obvious framework layer.

I am not necessarily advocating that we all work towards always creating domain-specific languages, but why doesn't the typical source code show SOME progression towards that extreme rather than seeming to always land squarely at the bottom of the levels of abstraction? As an industry, we certainly know how to do that, but why does it seem to not manifest in practice? Are we all just too busy, too rushed? Or is it something else?

Does your experience match mine at all?

Why do you think this tendency exists?

EDIT: I realize that I need to clarify an aspect of my experience

I have been a computer consultant since I was sixteen, which spans about 25 years. I have worked with many dozens of organizations spanning the public and private sector, lots of industries, and the gamut of sizes (mom & pop shops up to Fortune 100). I have worked with literally hundreds of programmers likewise ranging the gamut of newbie to twenty-plus years of experience. I have programmed in C, C++, C#, Java, VB (in all its variants), Python, Prolog, COBOL, Fortran, assembler, JOVIAL, JCL, SNOBOL, scripting languages galore, and many others. I have a similar breadth and depth as a DBA, sys admin, network admin, build engineer, architect, and much besides.

I can certainly vouch for the fact that this is NOT primarily an issue of years of experience. Some of the worst offenders that I have encountered are also the "most-experienced" in terms of years. As you can read in my edits below, I represent the other extreme of applying abstraction from day one. As some say: "Years make you older, but not necessarily wiser".

EDIT: some requested background

Yes, to answer a question, I can truthfully and accurately say that I have always practiced this kind of abstraction-building that I am advocating. I know that makes me very weird--an aberration even. Here is why that is so.

I grew up in a family of electricians and electronics engineers, building Radio Shack kits and playing with electronic components. I became interested in programming in 1978 at the age of twelve. I lived in rural Indiana (outside Richmond), and there were NO computers available to me. My school had none, and my parents could not afford the few personal computers available by then.

So, I began reading every book I could find that was even remotely related to computers, and I am a voracious reader. In three years, I exhausted my school library, the public library, and every other library within my parents' tolerance to drive me.

At that point, we moved to Fort Wayne and a school that had computers!!! Also, my dad acquired a Texas Instruments TI-99/4a and a Timex Sinclair 1000 (with 14K RAM expansion!) for my birthday. I continued reading every computer-related book I could find, but I also started programming in earnest.

Therefore, by the time I could actually touch a real computer, I had read hundreds of computer books about every aspect of programming, etc. I had also built rudimentary computers via Radio Shack kits and my own tinkering. By the time I actually starting programming, I definitely understood the value of abstraction.

Therefore, I readily admit that I am a mutant and I know that my experience is far from typical. Nevertheless, I don't think it qualifies me as somehow "special" in the grand scheme of things.

EDIT: some more requested background

Of my nearly thirty years, the vast majority have been as a consultant. I have had several stints as an employee, totaling several years, and I am an employee now at a Fortune 300 company serving as a development team manager and architect.

Answer 1

Experience (or lack there of)... Are you able to say that you've never done that before? I know I sure did a lot when I first begun programming. I use IoC (Inversion of Control) now, and don't understand how people could never use it, but that's sure a tough thing to bring to the table when someone is just getting a handle on the most basic of OOP principles.

Answer 2

It seems to me that this is most common with "undiciplined" developers. Once you get expierence (usually work experience) where your code is reviewed by others and they comment on the lack of brevety, and when you learn about better language features, and after you have been burned by making a piece of code like in your example and then add another field to the object and have to modify every if statement after the object is created, you learn to save yourself the hassle.

If you've got experienced programmers doing this, then you have a whole other problem. They probably just aren't that good at programing. That is, they can't raise their head up above the problem and see the larger impact of how the code is designed. In that case, you need to have them do more design ahead of time. Get them thinking "big picture" before they write code so that they don't forget while they write the code.

Answer 3

This is definitely not my experience, at least in developer written code. Much of this kind of stuff is done via code generation or library abstractions (.NET). On the project we are working on at the moment, there is actually very little coding being done to hook up a new feature, and we are using a N tie,r modular architecture (IOC, WebService, DAL).

I think it stems from people working in isolation, without being exposed to others who may have learnt some tricks or best practices. Also the presence of developers in a team who drive change is important (and who can design). That's the only way libraries, code generation and code patterns get created. The customer/manager/business does not understand these things.

Answer 4

I agree with Scott; Experience will allow a programmer to elevate their thinking and make the decisions about where an additional methods make sense or abstracts complicated logic. These decisions are very much a subjective call by the programmer or api designer.

One thing I find helpful to figure out where the right level abstraction belongs, is by thinking on a more abstract level of understanding the requirements, and using UML diagrams before gong right at the code.

Answer 5

Learning advanced programming paradigms such as Object Orientation, design patterns and even basic function encapsulation is mostly driven by need. Bear in mind there are plenty of programmers who are not professionals - they learned to program out of curiosity, for fun or to complete a small personal project.

Even "professional" developers may find themselves in a position that they know just enough to get their job done, and aren't motivated enough to advance their skills and knowledge.

Programmers who attempt to solve increasingly more complicated problems will inevitably try to find ways to reduce complexity and effort, and will learn about abstraction, encapsulation, code reuse, and common solutions to achieve those (design patterns). Of course it helps if they are a part of a team that has such developers already, as those will usually pass on their experience and knowledge even without any immediate need.

Unfortunately, it seems that most programmers (especially those who generate small scripts as you've mentioned) still belong to the first class. For most people, what works is good enough.

Answer 6

In my case I do this often, when I'm not yet sure what I'm supposed to do (with many problems one doesn't know the correct solution straight on) or when learning a new language or API, in these cases this is common, and healthy IMHO, because you actually begin to grok, not just become a casual user of the language, this way one can understand what things cost and other intricacies, however at a later point you SHOULD definitely refactor this sort of "monstrosity" away into a nice abstraction layer. So my conclusion here is "why don't more programmers refactor their code?"

Answer 7

The lower levels of abstraction provide some distinct benefits such as ease of understanding and accessibility for someone not already familiar with the code base. Moreover, lots of abstraction and encapsulation tends to be performed at the refactoring stage, rather than the RAD stage (this depends on the level of experience and individual programming tendencies), and project don't always get to the refactoring stage. The lower level abstractions can also provide insight into the construction and debugging process, which is probably the element most lacking in higher level abstractions and languages.

I also tend to write code in higher level abstractions more often than not, but there are definitely some trade-offs.

Answer 8

Problem is in nature of programming itself. Programmer should program! The more he thinks the less he programs. So once he starts to solve problem by coding he intentionally or subconsciously avoids any possibilities which may interrupt him and move away from final goal. As goal is working executable not design document result is very messy but quickly done code.

PS: I have 10 years of programming experience myself.

Answer 9

I totally agree with the previous posters that experience plays a major part here. Another factor that I've come across in this context is budgeting and estimation.

When a project isn't budgeted well or the problem domain isn't well understood by the developers, technical decisions tend to be made on the fly and things like proper abstractions tend to be left in the dust in the interest of making a short-term deadline.

A developer may be experienced enough to understand the need for abstractions, but not so experienced that such decisions can be made intuitively.

This may be a poor excuse for making bad long-term design choices, but I think it plays a bigger role than we'd like to admit.

Answer 10

When you see code snippets in answers on SO, the author has been given a keyhole view of the real problem domain. The result is bound to be a low level of abstraction. They don't know what abstractions are really appropriate, and I don't think you should take the resulting code as an indication of what their production code would look like if they took on the whole project.

So for example, if you ask someone "how would you test if two strings are both present", they're going to say "check them for null and empty". They aren't necessarily going to say, "write a function called isPresent that checks whether it's null or empty, and call it twice", even if that's what they'd actually do on a real project. In fact, on a real project they might decide that there's no such number as two, write something that checks whether a whole bunch of strings are all present, and call it with two strings, confident in the knowledge that somewhere down the line they'll need to check 3 and they'll be ready.

But to answer a simple question, you don't necessarily need to go into all that stuff. And the question you refer to wasn't "how do I refactor this code?", it was "how do I format this code?". In that situation, I think it's reasonable to talk about how one would, in general, deal with large boolean expressions, using the example given. That a large boolean expression is not in fact necessary in that case is true, and worth mentioning because it will help the questioner, but it's not the only valid topic of discussion. I'm sure some take the view that appropriate abstraction will always reduce all lines of code below 80 characters (80 having a special status which 68 and 12 don't have) such that the issue of how to format long lines can never arise, and answering the question in that vein is Just Plain Wrong. I don't share that view.

Nobody builds projects by combining snippets from SO (I hope). I think it goes without saying that they are just given for example purposes, are liable to be contrived to illustrate a particular point, and therefore don't always represent good design in other respects.

It's also worth bearing in mind that some people are brought up on extreme and agile methods (or at least have used them successfully), and will take the view that the first thing they think of that works is the right answer, until proven otherwise. In this view, if you have to think even for a moment about whether your abstraction should handle any edge cases that don't occur this time, but might in other hypothetical uses of the abstraction, then you're wasting your time. Abstractions would then be created when they're an essential part of the way the problem is stated, and as needed for reuse, or to break down a part of the problem which has become unwieldy. They would not be created in the anticipation that this will happen, and in particular would not be created for the sake of a couple of lines of example code.

Answer 11

When a programmer begins a project, the self-evident artifacts at hand are:

1. Some kind of definition of the problem to be solved (which is possibly expressed at some higher level of abstraction, if you can ferret it out, but it's not usually easy), and

2. Some kind of documentation for the tools to be used to solve the problem. And it's almost all provided (and documented, and examples shown) at the lowest level of abstraction.

Novice programmers immediately start to work with #2, with an intention that, with enough work along the shortest line between two points, they can put together something that lines up with #1. Basically, the #2 end of the gap is more confortable (and under their control, and easiest for them to understand) than the first. But, as we know, this straight-line brute-force bridge will involve much frustrating rework.

Over time, programmers learn to appreciate that some significant amount of time will in any case be spent structuring a better, more complete understanding of #1; and things end up better if you build abstractions out of the problem domain (which isn't under their control) that begin to fit into patterns that can drive programs.

A big part of maturing as a developer is learning to spend the time and effort to see the problem accurately, usually from a user's point of view; and then absorbing the patterns that will comprise what turns out to be the "obvious way to address such problems".

Sidenote: it's interesting how often users and developers manage to agree on requirements (and solutions) embodied in a spreadsheet => datagrid. Not that it's the best way, but it's probably the shortest path to a common understanding.

Answer 12

Rob Williams,

I'm pretty sure, if you're as nerdy as you claim to be, you've heard of the Sapir-Whorf hypothesis, which more or less asserts that a person's native language affects the way they understand and interact in the world. While this applies mostly to linguistics, many people can rightfully extend it to the world of programming languages: the languages that people code in affect the way they approach and solve problems. A good article on this topic is Paul Graham's essay Beating the Averages, and there are plenty of books with the title "Thinking in [C++ / Python / Perl / Language of your choice]".

Many imperative languages, especially C++, Java, and C# encourage programmers to write code in "bottom-to-top" style: programmers start "close to the hardware" and apply layers of abstraction as the need arises.

Have you ever seen the way code is written in functional programming languages, such as Lisp, Haskell, OCaml, F#, Erlang, etc. Programmers who write code in these languages approach problems from a different point of view: they start at 30,000ft and peel away levels of abstraction by necessity. Code tends to be more declarative in nature, resembling mathematical expressions, and it only gets close to the hardware when there is a specific requirement to optimize a tight loop.

Programmers adopt different programming styles because, if for not better reason, the languages they use pigeonhole them into one way of solving problems.

Answer 13

I can think of several reasons:

It begins with education. Having taught my share of software courses and dealt with enough students, I cannot tell you how frustrating it is that the introductory courses where these students learn to program focus on correctness and efficiency and not on structure. Once you've created a programmer who only cares about things working and running in the appropriate complexity class, you are going to have to work really hard to shift things.

I cannot tell you how many ways I've "approached" students on this issue to get the response: "but it works!".

Until the incentive system in schools leads to good development practices (e.g., refactoring), we're not going to get anywhere. Unfortunately, schools deal with pet projects that are not maintained over time.

My "evil dream" is to one day design an undergraduate curriculum where the student's capstone project required for graduation makes heavy use of the crappy code they've written as freshmen. Eventually, word will travel.

Of course, this problem is then manifested in the industry: your manager cares about immediate features and performance. Long term investments such as codebase readability and modularity are not as quantifiable, and have shared returns. You then have the "tragedy of the commons": why would I invest more at creating a decent codebase for the benefit of everyone when I am measured by my performance?

Answer 14

People want to avoid unnecessary effort. Which is more cheap? Cut'n'paste a piece of code which has done well for years or refactor an existing method so you can reuse common code?

The latter means to sit back, think, make a decision. The former is just a press of a key, an automatic gesture, which has been moved to the fingertips many years ago by constant use.

Things get worse when you're under pressure: Deadline, unfamiliar environment, whatever. The more stress, the more you tend to avoid extra effort and stick to the "old ways" even if they might eventually bite you ("but not today, I've already got enough on my plate, thank you very much").

Interestingly, the "old ways" are different for all of us. Some of us (like the Rob Williams) have learned to mind certain mistakes and even in the heat of the battle, five minutes before the ultimate deadline under enormous pressure to "just make it work", will avoid them. This is called experience or, as someone said, "something you should have had when you made it".

Unfortunately, it takes time to gather this experience and that's why you see so much of the same mistakes: Students aren't trained to avoid them (probably because of time constraints and all the other things they need to learn like integral math and signal theory) and so, when they get under pressure, they fall back to the patterns they installed in their brains at the time they started to learn how to program, that is when they had no experience at all.

Answer 15

The number one reason is a lack of experience in maintenance. Once programmers realise they are forced to trace back to five files and 8 function calls to find out why the current if() checked for == 2 and == 4 but didn't check for == 3, their own programming style changes too.

Answer 16

There's another side of the issue and I think we should understand both.

There's a benefit in using the lower level constructions. When you abstract something you hide details, create contract, use assumptions, and require context.

if(p == NULL || strlen( p ) == 0) /* I can see exactly what's being checked */

if( is_empty( p ) ) /* will it break on a NULL? */

Another issue is that once you've created a function, less experienced programmers will just use it, often without real understanding of what the side effects will be. The is_empty() above is, probably, too trivial for misuse.

Being a lead engineer, I've had numerous times when a programmer would say, when presented with a defect that he created, "yes, but I just used your function!". Yes you did, but did you also read the contract? Did you understand all the pre- and post conditions? Often people will just copy the code that looks like it will do the work.

The definite moment for me to create a higher level abstraction is when 1) a function exceeds a single screen or 2) the code starts to be copied and it is more than a line.

Another trigger for an abstraction will be to wrap up something tricky or sophisticated.

The testing for an empty string above doesn't trigger any of these with me.

Answer 17

It's a power vs responsibility issue.

Programmers that aren't primarily architects tend to think of power, they want to see everything in their routines even if it does mean doing bit shifts in a high level service. They like the power because they like to see exactly what they're doing.

Programmers that are looking at the system as a whole generally want to give up power and responsibility and are happy to include an IsPresent() method, heck they don't even mind making it virtual and just want to define roughly what it should represent.

We have a very interesting mix in my team and I don't necessarily think that is always a bad thing (aside from the odd fire due to this kind of thing). However when when I'm looking at the code I'm hoping that as the layers rise the amount of options declines, simplifying things (as per your IsPresent() method) so the top, top, top layer of code reads well and is very simple.

Answer 18

Maybe I'm missing something here,

but it looks like the person who wrote this was making the point that this is the only order in which you should perform these tests... if your object o is null then any of the other tests would cause this code to generate some kind of exception/crash (depending on what language we're in). So that one HAS to be done first and the example was probably written with the intention of making this obvious (if not, the author is in for a surprise if the evaluations are ever re-ordered!).

This in and of itself doesnt make it bad code... maybe this is the only place in the application that this kind of test is performed?

If you step back and see that the application is doing this in ten other places THEN maybe it is time to say 'we want to code this as a function so we dont have 10 identical pieces of code to maintain in our application'.

The level of abstraction provided can sometimes make things MORE confusing by making you jump around in the code just to see what is going on. As others stated sometimes this is not the best scenario especially on SO when you are trying to explain your answer in the clearest possible way (which may not always be the most computationally efficient or maintainable code!)

That stuff I wrote in the second paragraph, would it be easier or harder to understand if instead of 0== and NULL== we had isThisOk(obj) and willThisBugout(obj) in the code. What was the author trying to explain? Will this code work properly? I dont know, let me go check what that function does... what file is it included in?

Abstraction is one of our most powerful tools/concepts, but if we are operating ON the level and not above it, abstraction may do nothing but add complexity. Abstraction is our tool to FIGHT complexity and the more complexity is present in our code, the more apparent it will be as to where we need to use it. In a 5-line example program, it may not always be painfully obvious what needs simplifying :)

If I'm saving my word processing document to disk, I'm glad I can just File.write(allMyData) and not worry about what cylinder, head, sector the drive is pointing to, because at this level of abstraction it is not relevant to the task. If I were writing hard drive firmware however that might be a different story.

Edit - I love n-alexander's answer up there.... "When you abstract something you hide details, create contract, use assumptions, and require context."

There you go :)

Answer 19

i have been programming for more than 40 years. i have taught some college graduate and undergraduate courses as well as some commercial ones. abstraction is just something than many people find difficult. it's like seeing algebra when you have been doing arithmetic.

another reason is that most programmers think that they do not have the time to refactor or clean up their code. so the code tends to just grow in place and become fragile.

there is also a fud factor. the more uncertain one is about ones or other peoples code, the more concrete one tends to be.

Answer 20

One problem is that new programming languages come so fast and they replace the previous ones. Just when we get something to at least work, we move to the next language and write new applications from the start again. The new language might be a bit higher in the abstraction, but we still create database access, user interfaces, business logic, and so on. There are also platforms and frameworks that add stuff to the code instead of removing.

Sure some progress has been done and some programmers care about abstraction, but things need to improve more.

Answer 21

I would of suggested to write an extension that is available for all objects :)

public static bool IsDefaultOrNull(this object value)
{
    if (value is string)
      return string.IsEmptyOrNull(value as string);
    return value == null;
}

or ->

public static bool IsDefaulted<T>(this T value)
{
    if(value is string)
       return string.IsEmptyOrNull(value as string);
    return value == default(T);
}

... just a sidenote :P

Answer 22

Of your 25 years were you primarily an employee or a consultant?

My limited experience in being both, I can say that I usually follow more best practices when I'm a consultant. I'm not exactly sure why this is but I can take a few guesses. As an employee, I'm responsible for my code indefinitely. In light of that, design and structure don't end up being nearly so important to my supervisors as meeting deadlines. As a consultant, I'm aware that once I leave my clients are going to be looking over my code. If they hear that the design is confusing and the structure is... non-existent odds are I'm not going to get any repeat business. Also as a consultant the project feels new and fun and my creative juices are flowing and I remember why I wanted to be a developer. As an employee I'm either cleaning up an existing application made by another employee who is long gone or I end up designing report after report for the same application year after year.

Who's to blame for this is depends on who you're asking I suppose. Is it supervisors that have no experience with or understanding of application development? Or am I just being lazy because I can just get away with it?

Regardless, I think that's the answer to your question. They're employees, not consultants.

P.s. This isn't to say that there aren't any decent employee developer positions. My current employer is by far the best I've worked with so far. But by and large, I think this accounts for most of what you've observed.

Answer 23

I know what you mean.

I actually cringe when I hear the words "levels of abstraction" as if "abstraction" is somehow something glorious (if poorly defined).

I would rather appeal to the DRY (don't repeat yourself) principle, i.e. make the code cleaner and less wordy and repetitious, and easier to change. Writing and using functions/classes are a way to do that.

That is a skill that not everyone uses as much as they could.

Answer 24

To loosely answer your question, I too am very disappointed when I see people writing (and then copying and pasting) code very similar to your example.

I am surprised that people don't commonly make use of inbuilt String (and other data type) functions (like String.IsNullOrEmpty or Boolean.Parse in .Net). All these are well documented, and will save you time and time again from If str IsNot Nothing AndAlso str IsNot String.Empty (or sometimes, <> "", urgh!)

...but knowing about these functions takes experience, and so, unfortunately, as the Junior Developers have to write applications and bug-fixes for Production in much the same way as Senior ones do, I don't think this is a problem that won't easily be solved.

Everyone has to learn some way, mine was through books/Uni and a couple of major Production outages as a result of my code... Others (like the OP) might have taken a different path, but we (should, hopefully) all get there in the end.

Answer 25

It seems appropriate to assemble a collective answer from everyone's responses, since they now seem stable. Here it is.

Abstraction is always undesirable

Some people actually see abstraction as undesirable--a waste of effort. For example, students being introduced to abstraction by a teacher reviewing their code often respond "but it works". Even people programming for a living often don't see a need; their code is good enough. However, Martin Fowler has written a wonderful book on refactoring that I consider essential reading for all programmers. He does a great job illustrating why refactoring, which is a key step to obtain abstraction, is practical and desirable in nearly all cases. Therefore, to answer the begged question, "abstraction is desirable" is considered an axiom.

Abstraction is undesirable in exceptional cases

On the other hand, nearly every rule has an exception, and some have explained that abstraction can rightly be seen as undesirable in some cases. For example, in accordance with YAGNI, abstractions should be just-in-time (JIT), so a premature abstraction can be worse than no abstraction. Martin Fowler would probably suggest that this sentiment only applies in the face of a "big bang" abstraction, and he would probably recommend smaller refactorings that create the abstraction in smaller chunks. However, also in keeping with YAGNI, some argue that one should create abstractions only when the code begins to smell in certain ways, such as: when a method exceeds one screenful, when you want to copy more than one line for reuse, or when you feel the need to isolate something tricky or sophisticated. Another common exception occurs when one is learning an API, where uncertainty leads to concreteness that should be refactored later.

n-alexander beautifully described the consequences of creating an abstraction, in that it hides details, creates a contract, uses assumptions, and requires context. Therefore, creating an abstraction implies taking on significant responsibility that should not be taken lightly.

Indeed, many people recommend considering the potential audience carefully. They observe that the use of abstraction can make code less readable in the short-term--one may have to hunt through various source files to find definitions for numerous parts and participants in an abstraction. Some readers may even have to learn new (English, technical, etc.) words introduced by the abstraction. Therefore, lower-level abstractions are more accessible and universal, especially the ones delivered with a particular development platform. Failure to consider the audience may result in misuse of an abstraction because users ignore its implications, resulting in them placing the blame for any defects on the writer of the abstraction!

There is a lack of time for creating abstractions

As was mentioned in the question, sometimes we know better but we just don't have the time to create those abstractions or, at least, we don't feel that we do. We intend to refactor and to create those abstractions, and we may even know what they should look like, but we don't do it, yet.

Sometimes we feel the rush towards the next looming release (tight deadline). Perhaps it may be lack of budget, understaffing, the impact of providing support, that leaves us feeling that abstraction must be put off for now. Perhaps it is some fear, or misconception, that makes us think that abstraction is expensive in time and requires special planning and treatment.

Some have proposed that, under that pressure, we tend to fall back on practices that we learned when our experience was zero. We drop all our best practices, we temporarily forget the advanced techniques, and we just grind away at the most concrete level.

In an interesting twist, I have found that the most common occasion for me to experience this phenomenon occurs at the introduction to a new development language/platform/environment. I begin programming and wish that I had all my favorite abstractions, but I am overwhelmed in the face of the effort to recreate them. Of course, my new development landscape may include language features and APIs that already incorporate some of my favorite abstractions or make them unnecessary, but it will take me time to discover that. I usually begin recreating the essential abstractions as I create my first few projects, but I find myself wishing that I had the means to instantly "generate" them in my new development landscape. Indeed, I have repeatedly considered starting a project to do exactly that--express my favorite abstractions in a neutral way that can be "generated" into each new development landscape that I encounter, without a lot of fuss and effort. It could happen, I think...

There is a lack of experience for creating abstractions

Now we get to the heart of the controversy--that many programmers lack experience with creating abstractions, to the point of being unwilling or unable to do so effectively.

Some have argued that our institutionalized educational system is largely to blame, which perhaps focuses on correctness and efficiency to the exclusion of design and structure, resulting perhaps in the common claim that abstraction is generally undesirable and unnecessary. This may particularly reflect the fact that most educational processes consist of writing a series of unconnected, trivial programs that lack the dynamics of the real world that suggest, encourage, or even force the use of abstraction. The artificial environment lacks real users, lacks real use, lacks maintenance, lacks scale, and often even lacks peer review.

Even in the real world of programming, beyond the educational atmosphere, programmers are often isolated. They may be the sole programmer in their organization, or at least on their applications, or they may have peers but no one that is able and willing to serve as a mentor. The organization may simply push too hard and fast to allow for any kind of mentoring, peer review, or learning of any significance.

Maybe a particular programmer has yet to perform any software maintenance, or receive any feedback from someone attempting to maintain their code. Since abstraction is driven largely by the need for readability to support maintenance and reuse, the lack of that particular kind of experience can easily leave one questioning the value of such preventative measures.

Most programmers have only ever worked in-the-small, meaning that the scale of their applications/systems is on the low end of the spectrum. It is a simple fact that programming in-the-large is relatively rare, since there are few organizations that have the need and/or resources to reach that scale. Much of the drive for abstraction, and design patterns, and many other advanced programming approaches comes from the special considerations of programming in-the-large. Simply put, scale changes the dynamics of everything. It seems to me that we should make a point of differentiating our advice based on scale, since most of our advanced techniques are optional or irrelevant when programming in-the-small. However, abstraction seems to be relevant at practically every scale.

Unfortunately, our industry is focused on tools rather than technique. Whether in the educational process, the hiring process, the training process, the book-writing process, even the product evaluation process, everyone seems focused more on the tools than on the application of the tools. It seems that the situation may be improving, though.

As a reflection of this situation, it seems that novices tend to work from the tool (no abstraction) up to the problem (high abstraction, maybe), but experts tend to do the opposite. It also seems that novices want to hold onto power and responsibility by seeing low-level code on the screen, but experts let it go to achieve brevity and simplicity through abstraction.

Another consequence of this situation is that many programmers are never exposed to various abstraction principles that make implementation easier or even possible. Many have never heard of design patterns such as Inversion of Control or modularity approaches such as functional composition. Hence they are unsure or ill-equipped to introduce the abstractions that would help them.

Someone cited the Sapir-Whorf hypothesis which says that a person's thoughts are shaped and constrained by the language that they use, with particularly interesting application to computers. So, perhaps, the lack of abstraction that we see comes in part from a large number of people being exposed only to imperative languages (e.g., C) that encourage low-level, bottom-up, procedural code. Meanwhile, only a few are exposed to functional languages (e.g., Lisp) that encourage high-level, top-down, declarative code.

There is a complete breakdown in creating abstractions

It is apparent, just as should be expected from a decent grasp of human nature, that there are many people who don't care: they are just not willing. They understand the concept of abstraction, maybe they see its value, but they won't invest themselves in doing it.

For some, perhaps it is a lack of discipline. They want to invest themselves, but they can't quite pull it off consistently. Perhaps the organization puts obstacles in the way, probably unknowingly, and the programmer lacks the desire to overcome them. There may be a variety of reasons, but what matters is that it does not happen.

For others, perhaps it is a lack of professionalism. They see the value, have the discipline to execute, but choose not to do so. Maybe they do abstraction for their own projects, but not at work. Maybe they recognize that abstraction is not part of their performance review. Maybe they don't care if it will improve the company's bottom line, since that doesn't benefit them. In whatever manifestation, they put themselves and their own interests before their customer, so abstraction does not happen. Some argue that this is a key difference typically between employees and consultants.

Finally, to complement those that are not willing, it is apparent that some people just don't get abstraction: they are just not able. As a matter of fact, there are many articles on the 'net and elsewhere that acknowledge the inability of many people to get programming at all. I am not trying to claim that such people are less valuable (I vehemently don't believe that), but simply that programming, and particularly abstraction, is similar to painting or dancing or singing or any other challenging activity--some people have the talent, some people get it, and the rest of us simply don't.

Some might find this example pathological, but I once had a development manager that strictly forbade me from creating abstractions because it resulted in "too many files in source control". Someone else on the development team (about a dozen members at that point) had apparently complained that all those files bothered them in some way, although I was never given an explanation. As the team lead and architect, I had spent months alone fleshing out a design in the newly-minted C# on .NET 1.0 (mid-2002). I had created trivially-simple, even obvious, abstractions such as EmailAddress, PhoneNumber, CustomerName, etc. Now that we had ramped up a development team, I was merely continuing to occasionally add new but comparable abstractions as we encountered the need. Each was supported by a solid set of unit tests (over 4,500 in all) that verified their functionality and illustrated their use. Apparently, much of the team did not get or want abstraction, which was further apparent when reading their code, since that code perfectly illustrated this question.

When I was younger, I had assumptions that someone who didn't get abstraction, or didn't get programming, or didn't get troubleshooting, would recognize that fact and would act accordingly (meaning to seek improvement or seek to do something else). However, after much experience, and particularly after watching TV shows like "American Idol", I have since concluded that many people simply don't get that they don't get it. Furthermore, their family, friends, and coworkers often seem to help perpetuate that ignorance (making them accessories).

Thankfully, with singing, dancing, and many other activities, it is painfully obvious to many of us when someone doesn't get it. But apparently not so with programming (especially abstraction)--the intangible, invisible, generally incomprehensible nature of that activity seems to make detection difficult, and to make description very much more difficult still. This is illustrated by many interesting articles about interviewing programmer candidates, where I have seen estimates that only one in two hundred applicants could actually complete a trivial programming task.

Therefore, I really try to examine my own thoughts, abilities, and actions, so that I might save my fellow humanity from the pain of watching me when I don't get something. And I promise never to apply to "American Idol".

Answer 26

Unix and Object-Oriented Languages answers a subset of your question corresponding to OO abstractions:

...

We observed above that the Unix tradition of modularity is one of thin glue, a minimalist approach with few layers of abstraction between the hardware and the top-level objects of a program.

...

OO languages make abstraction easy — perhaps too easy. They encourage architectures with thick glue and elaborate layers. This can be good when the problem domain is truly complex and demands a lot of abstraction, but it can backfire badly if coders end up doing simple things in complex ways just because they can.

...

As the author of The Elements of Networking Style once observed in a slightly different context [Padlipsky]: “If you know what you're doing, three layers is enough; if you don't, even seventeen levels won't help”.

...

A hammer and general-purpose tool-building factory factory factory nicely illustrates the above points:

"So this week, we're introducing a general-purpose tool-building factory factory factory, so that all of your different tool factory factories can be produced by a single, unified factory. The factory factory factory will produce only the tool factory factories that you actually need, and each of those factory factories will produce a single factory based on your custom tool specifications. The final set of tools that emerge from this process will be the ideal tools for your particular project. You'll have exactly the hammer you need, and exactly the right tape measure for your task, all at the press of a button (though you may also have to deploy a few configuration files to make it all work according to your expectations)."

"So you don't have any hammers? None at all?"

"No. If you really want a high-quality, industrially engineered spice rack, you desperately need something more advanced than a simple hammer from a rinky-dink hardware store."

See Apache XML-RPC framework for another illustration.