Is monkeypatching stdlib methods a good practice in Python?

Question

Over time I found the need to override several stdlib methods from Python in order to overcome limitation or to add some missing functionality.

In all cases I added a wrapper function and replaced the original method from the module with my wrapper (the wrapper was calling the original method).

Why I did this? Just to be sure that all the calls to the method are using my new versions, even if these are called from other third-party modules.

I know that monkeypatching can be a bad thing but my question is if this is useful if you use it with care? Meaning that:

• you still call the original methods, assuring that you are not missing anything when the original module is updated
• you are not changing the original "meaning" of the methods

Examples:

• add coloring support to python logging module.
• make open() be able to recognize Unicode BOM masks when using text mode
• adding logging support to os.system() or subprocess.Popen() - letting you output to console or/and redirect to another file.
• implementing methods that are missing on your platform like os.chown() or os.lchown() that are missing on Windows.

Doing things like these appear to me as decent overrides but I would like to see how others are seeing them and specially what should be considered as an acceptable monkeypatch and what not.

Answer 1

None of these things seem to require monkeypatching. All of them seem to have better, more robust and reliable solutions.

Adding a logging handler is easy. No monkeypatch.

Fixing open is done this way.

from io import open

That was easy. No patch.

Logging to os.system()? I'd think that a simple "wrapper" function would be far better than a complex patch. Further, I'd use subprocess.Popen, since that's the recommended replacement.

Adding missing methods to mask OS differences (like os.chown()) seems like a better use for try/except. But that's just me. I like explicit rather than implicit.

On balance, I still can't see a good reason for monkeypatching.

I'd hate to be locked in to legacy code (like os.system) because I was too dependent on my monkeypatches.

---

The concept of "subclass" applies to modules as well as classes. You can easily write your own modules which (a) import and (b) extend existing modules. You then use your new modules because they provided extra features. You don't need to monkeypatch.

even if these are called from other third-party modules

Dreadful idea. You can easily break another module by altering built-in features. If you have read the other module and are sure the monkeypatches won't break then what you've found is this.

1. The "other" module should have had room for customization. It should have had a place for a "dependency injection" or Strategy design pattern. Good thinking.

2. Once you've found this, the "other" module can be fixed to allow this customization. It may be as simple as a documentation change explaining how to modify an object. It may be an additional parameter for construction to insert your customization.

3. You can then provide the revised module to the authors to see if they'll support your small update to their module. Many classes can use extra help supporting a "dependency injection" or Strategy design for extensions.

If you have not read the other module and are not sure your monkeypatches work... well... we still have hope that the monkeypatches don't break anything.

Answer 2

Monkeypatching can be "the least of evils", sometimes -- mostly, when you need to test code which uses a subsystem that is not well designed for testability (doesn't support dependency injection &c). In those cases you will be monkeypatching (very temporarily, fortunately) in your test harness, and almost invariably monkeypatching with mocks or fakes for the purpose of isolating tests (i.e., making them unit tests, rather than integration tests).

This "bad but could be worse" use case does not appear to apply to your examples -- they can all be better architected by editing the application level code to call your appropriate wrapper functions (say myos.chown rather than the bare os.chown, for example) and putting your wrapper functions in your own intermediate modules (such as myown) that stand between the application level code and the standard library (or third-party extensions that you are thus wrapping -- there's nothing special about the standard library in this respect).

One problematic situation might arise when the "application level code" isn't really under your control -- it's a third party subsystem that you'd rather not modify. Nevertheless, I have found that in such situations modifying the third party subsystem to call wrappers (rather than the standard library functions directly) is way more productive in the long run -- then of course you submit the change to the maintainers of the third party subsystem in question, they roll your change into their subsystem's next release, and life gets better for everybody (you included, since once your changes are accepted they'll get routinely maintained and tested by others!-).

(As a side note, such wrappers may also be worth submitting as diffs to the standard library, but that is a different case since the standard library evolves very very slowly and cautiously, and in particular on the Python 2 line will never evolve any longer, since 2.7 is the last of that line and it's feature-frozen).

Of course, all of this presupposes an open-source culture. If for some mysterious reasons you're using a closed-source third party subsystem, therefore one which you cannot possibly maintain, then you are in another situation where monkey patching may be the lesser evil (but that's just because the evil of losing strategic control of your development by trusting in code you can't possibly maintain is such a bigger evil in itself;-). I've never found myself in this situation with a third-party package that was both closed-source and itself written in Python (if the latter condition doesn't hold your monkeypatches would do you no good;-).

Note that here the working definition of "closed-source" is really very strict: for example, even Microsoft 12+ years ago distributed sources of libraries such as MFC with Visual C++ (as their product was then called) -- closed-source because you couldn't redistribute their sources, but still, you DID have sources at hand, so when you met some terrible limitation or bug you COULD fix it (and submit the change to them for a future release, as well as publishing your change as a diff as long as it included absolutely none of their copyrighted code -- not trivial, but feasible).

Monkeypatching well beyond the strict confines within which such an approach is "the least of evil" is a frequent mistake of users of dynamic languages -- be careful not to fall into that trap yourself!