Motivation for Overengineered APIs?

Question

It seems like there's a decent amount of dislike around here for horribly overengineered APIs that are designed to be infinitely flexible and thus don't make simple things simple. Nonetheless, it seems like there is no shortage of APIs that require you to use 8 different classes and write 20 lines of boilerplate just to accomplish simple, common tasks. I won't mention names because this isn't supposed to be a flamewar about whether specific APIs are overengineered.

What do you believe is the root cause of these horribly overengineered APIs? What do you think needs to happen to prevent API designers from creating such monstrosities?

Edit: IMHO, not even the creation of reusable code is really a good answer because if the API is ridiculously difficult to use and requires tons and tons of boilerplate, the benefits of reuse become questionable.

Answer 1

"Premature optimization is the root of all evil."
--Don Knuth

Even so, it is very, very tempting because programmers instinctively like efficiency and cleverness.

Answer 2

Second system effect?

Answer 3

I don't think anything can prevent people from overthinking problems. It's inherent in problem solving. Methodologies like XP try to discourage it, but when it comes down to it, everyone thinks "But if I make this more generic, then I can reuse it in such and such"

Answer 4

The root of all evil is developers not being a) smart/experienced and b) adequate enough.

Answer 5

I believe this is often a consequence of the so-called Second System Effect. Designers take the lessons learned from their first cut of the "version 1" design, and make the next version so much more flexible that it becomes overengineered and hard to understand.

Fred Brooks' book The Mythical Man-Month introduced this term and talks about it in detail.

Answer 6

I'm not sure if these are a case of over-engineering or not enough abstraction. The windows api is a prime example of this.

Once upon a time I spent a good deal of time writing a print and preview engine. I had to decode the windows api calls needed to display things on the screen and output to a printer. In creating the api abstraction i tried to think in terms of what the developer is trying to accomplish ... for example: "I want to draw a 1point wide, red line from coordinates (1, 1) to (8, 1) - expressed in inches."

the equivalent windows api for this involves many, many irritating lines of code ... create a brush, select it into a device context, set the starting point, handle conversions from inches to pixels, draw to an end-point, etc. my abstracted api is a single call: dpLine(documentHandle, x1, y1, x2, y2, width, color); // where x1, x2, y1, y2 are expressed in inches

In this case, I think the windows gdi api is just too low level. I'm sure there's good reasons for the things they've done and just didn't have time/energy to make a proper interface for programmers that are likely to use it. The reason for the monstrosity is probably just time deadlines. The api is technically accurate; it allows a programmer to do what they need. That is good enough to ship it. But, it is so low level that 3rd party abstractions are necessary to make it usable. IMO, you can make an argument for an OS to provide a low-level, complicated api like this, but a 3rd party tool shouldn't be that complicated.

-Don

Answer 7

I think Python suffered from Second System Effect. In version 2.x two types of classes and different semantics exist.

Hopefully Python 3.0 solves most of this.