Learning to write reusable libraries

Question

We need to write simple scripts to manipulate the configuration of our load balancers (ie, drain nodes from pools, enabled or disable traffic rules). The load balancers have a SOAP API (defined through a bunch of WSDL files) which is very comprehensive but using it is quite low-level with a lot of manual error checking and list manipulation. It doesn't tend to produce reusable, robust code.

I'd like to write a Python library to handle the nitty-gritty of interacting with the SOAP interface but I don't really know where to start; all of my coding experience is with writing one-off monolithic programs for specific jobs. This is fine for small jobs but it's not helping me or my coworkers -- we're reinventing the wheel with a different number of spokes each time :~)

The API already provides methods like getPoolNames() and getDrainingNodes() but they're a bit awkward to use. Most take a list of nodes and return another list, so (say) working out which virtual servers are enabled involves this sort of thing:

names = conn.getVirtualServerNames()
enabled = conn.getEnabled(names)
for i in range(0, len(names)):
    if (enabled[i]):
        print names[i]
conn.setEnabled(['www.example.com'], [0])

Whereas something like this:

lb = LoadBalancer('hostname')
for name in [vs.name for vs in lb.virtualServers() if vs.isEnabled()]:
    print name
www = lb.virtualServer('www.example.com').disable()

is more Pythonic and (IMHO) easier.

There are a lot of things I'm not sure about: how to handle errors, how to deal with 20-odd WSDL files (a SOAPpy/suds instance for each?) and how much boilerplate translation from the API methods to my methods I'll need to do.

This is more an example of a wider problem (how to learn to write libraries instead of one-off scripts) so I don't want answers to these specific questions -- they're there to demonstrate my thinking and illustrate my problem. I recognise a code smell in the way I do things at the moment (one-off, non-reusable code) but I don't know how to fix it. How does one get into the mindset for tackling problems at a more abstract level? How do you 'learn' software design?

Answer 1

If you are not afraid of C++, there is an excellent book on the subject called "Large-scale C++ Software Design".

This book will guide you through the steps of designing a library by introducing "physical" and "logical" design. For instance, you'll learn to flatten your components' hierarchy, to restrict dependency between components, to create levels of abstraction.

The is really "the" book on software design IMHO.

Answer 2

There's an excellent presentation by Joshua Bloch on API design (and thus leading to library design). It's well worth watching. IIRC it's Java-focused, but the principles will apply to any language.

Answer 3

"I don't really know where to start"

Clearly false. You provided an excellent example. Just do more of that. It's that simple.

"There are a lot of things I'm not sure about: how to handle errors, how to deal with 20-odd WSDL files (a SOAPpy/suds instance for each?) and how much boilerplate translation from the API methods to my methods I'll need to do."

1. Handle errors by raising an exception. That's enough. Remember, you're still going to have high-level scripts using your API library.

2. 20-odd WSDL files? Just pick something for now. Don't overengineer this. Design the API -- as you did with your example -- for the things you want to do. The WSDL's and the number of instances will become clear as you go. One, Ten, Twenty doesn't really matter to users of your API library. It only matters to you, the maintainer. Focus on the users.

3. Boilerplate translation? As little as possible. Focus on what parts of these interfaces you use with your actual scripts. Translate just what you need and nothing more.

An API is not fixed, cast in concrete, a thing of beauty and a joy forever. It's just a module (in your case a package might be better) that does some useful stuff.

It will undergo constant change and evolution.

Don't overengineer the first release. Build something useful that works for one use case. Then add use cases to it.

"But what if I realize I did something wrong?" That's inevitable, you'll always reach this point. Don't worry about it now.

The most important thing about writing an API library is writing the unit tests that (a) demonstrate how it works and (b) prove that it actually works.