Interesting Python system utilities you have made?

Question

I am slowly but surely teaching myself Python. I learn best by doing. I'm looking for some neat system productivity kind of procedures that I could try making that you have found useful for yourself. Some of the modules I've successfully made and use are like these:

• Zip a folder
• Zip a whole set of folders to an archive as an automatic backup
• App launcher that opens my most frequently used programs at startup

But I'm kinda stuck now. What else could I do?

Answer 1

I made a snippet that listed the total, remaining and used space of each hard drive on the system in neat columns. Needed it for reporting uses.

Also made an email log parser, which was a quite funny exercise. I had not used custom state machines prior to that very much, but this got me really interested in their potential applications (pretty much everything hehe.)

Answer 2

How about something that looks at the magic numbers of all the files in a given directory and fixes the file extension if necessary. For example, if a .wav file starts with "MThd" it's probably a Midi file that somehow got misnamed.

Answer 3

I used Python as a simple stats script (things like battery, cpu usage, memory) to pipe into the dynamic bar in dwm.

Similarly as a backend to Openbox's pipe menus.

Answer 4

There are many custom log analyzers, installers, version checkers etc. Some scripts gather data for visualization in tools like Cacti or MRTG.

You can find mamy system utilities written in Python at Python Recipes

Answer 5

I have lots of songs, videos and images which I dump into folder named very intuitively "myfolder", i try to be organize but with time I have found many duplicate songs in various folders, same images at multiple places and so on

so i have written a script which i run to delete duplicate files, that could be a interesting exercise.

SO question http://stackoverflow.com/questions/748675/finding-duplicate-files-and-removing-them does implement similar script :)

Answer 6

I once wrote a small Python module that I integrated with the "cleanup" utility menu on WinCVS. It provided a convenient shortcut to easily remove all those files that Microsoft Visual Studio generates per project that should not be put under version control but were somehow committed in.

Answer 7

Have a look at the book Python for Unix and Linux System Administration. You will probably find some inspiration there.

Answer 8

The two self-made tiny text-utilities I use the most are "sum" and "checkUTF8". The first is a filter that sums up the lines in a file on standard input, like so:

$ grep '[0-9.]' somefile | sum
567567

... needless to say, I do alot of exploratory log-crunching.

The second checks whether at text-file is properly encoded in UTF8 and if not, shows the line number, byte offset, character offset and actual text of the problem line, with the start of the problem area highlighted. This makes it possible to correct the problem with a hex-editor, if necessary. It is much less of a problem now than when I converted from iso8859-1 to utf8, but it still happens.

Though "sum" is easily made in any scripting language you need good unicode-support for the other one.

Answer 9

Try writing a very simple e-mail client, as a step toward being able to e-mail your computer specialized messages that tell it to do specific things, like run a job, reboot, reply with some information, etc.

Answer 10

On the Windows platform, I wrote a wrapper class around the win32file.FindFilesW() API (from the python extensions for Windows package) to efficiently calculate the total size of any given folder, local or remote. The API is itself a wrapper around the FindFilesW() function from the Win32 API. For my class object, I added additional features that were very helpful beyond just getting the total folder size. In particular, I added a feature to capture the full path of files and sub-folders that were denying access so that I could know whether I was getting a skewed or true result; I added a feature to retrieve the total size up to a certain amount (in case I only wanted to know whether any data existed at all, e.g. 1 byte, 15 bytes, 1MB, etc.); and I added a feature to continue on access denied errors. I did these because in VBScript (which I originally used, for political reasons), the ability to capture the total size of a given folder will fail if it encounters any access denied errors. And in the Windows Explorer shell, looking at the properties of a folder will tell you the total size, but not notify you if any access denied errors were encountered, thus potentially skewing the size property without you knowing.

Answer 11

Another idea: Read The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) from "Joel On Software" and then write a utility to scan all the HTML files on your system and make sure they are encoded correctly. That is, make sure that the content-type the page claims to have is the same as the one that was actually used.

Answer 12

I also wrote a PyZip utility to do quick backups. It takes a filename template and a list of file patterns, something like this:

pyzip store\ProjectFiles-[date] *.py *.txt

if the "store" folder doesn't exist, it is created

if the template filename doesn't include ".zip", it is added

a [date] token is replaced with the current date, 20090624

likewise for [time], 155859

I'm working on a [#] token to be replaced with the next matching number based on existing filenames matching the whole pattern, for numbered instead of date-stamped backups

as time goes on, I'd like to replicate as much of the old PKZip command line capability as possible, and create a corresponding PyUnzip

---

I generally use ActiveState Python on Windows machines, and this comes with the win32 extensions, so I have been writing lots of ActiveDirectory management scripts, Word and Excel COM automation, managing small SQLite database, and lots of other things that I used to do with VBScript, and many things that couldn't be done (easily anyway) with VBScript.

Answer 13

I wrote a script that uses rsync to backup a remote directory tree on a local drive, and then starts rotating the backups after an arbitrary number of runs using only the deltas (sorta like time machine on the mac). Got that sucker running on a few geographically separate machines and feel much better about the safety of my data from destruction at least.

Large parts of the script were ported from other people's bash scripts, so that provided a nice stable feature scope to crib off of - maybe taking a look at some utility shell scripts would give you an idea to "translate" it into python, thus making it a little more extensible and readable.

You can check out the repo on bitbucket.

Answer 14

I wrote a util to rsync, set user/group and normalize my music collection. Yeah it could have been done in bash but I suck at bash.

I sort of a continuous integration process for some javascript code at a former employer that worked off of post commit hooks in mercurial to do various operations like minify/compress run YUItest and other steps that were needed to get it ready for production use. I doubt they are still using it though. It was built in an attempt to save me some frustration.

More recently I built a script to rip through a bunch of jwc's and generate png's via graphviz showing the dependencies between all the tapestry components in my current employers code base. We will probably be working that into the CI for our nightlies at some point.

And lastly, about 3 years ago, I replaced about 10K lines of really nasty t-sql with what ended up being a full blown py-tk app for migrating customers data from version of the app to another.

There's been various utils to parse and pull various things out of server logs in a pinch.

other fun stuff.

Answer 15

Using Python 2.5 I wrote a script that dumps my Subversion repository, compresses the dump file and scp's it over to two backup servers. Email me with the results of the backup. This is done on a daily basis. Having done this I realised how much I like Python and how powerful it can be.

The next step is to modify the script to use rsync and copy over some web site backups as well.

Certainly showed me the utility of Python.

Answer 16

I wrote a script to find all the jar files on my system and build a java classpath environment variable to set when my shell starts up interactively. That one script saved me many hours of problems.

When I needed to use a new jar, I just downloaded it, and reran the script.

I wrote a related script that let me search for a particular Java class name/file/etc in all of the jars on my system.

In general, find the simple repetitive tasks that you do each day and write a script to do them. You'd be amazed how much time you can save, and how much you'll learn.

For example, it's surprisingly easy to write a program that puts an icon in the system tray that changes according to whether you have any new unread emails.

Answer 17

I wrote a script that uses my cellphone's bluetooth ID as a sort of physical security token. When the PC (Windows) resumes from sleep, it scans for nearby Bluetooth devices, and if it doesn't locate my phone, locks the workstation.

Corny, and not very secure, but serves its purpose.

"""
Intended to run on Windows resume. Scans for bluetooth devices and if a 
particular device is not present, locks the computer.
"""
import bluetooth
import ctypes
import sys

# Bluetooth UID of "token" device
wanted = "XX:XX:XX:XX:XX:XX"

print "performing inquiry..."

# Hack.... if 0 devices are present, pybluez throws an exception
passed = False
try:
    discovered = bluetooth.discover_devices()
    if wanted in discovered:
        print "Found phone"
        passed = True
    else:
        passed = False
except:
    passed = False

if not passed:
    print "Locking"
    ctypes.windll.user32.LockWorkStation()

Requires:

1. pybluez for bluetooth
2. Hibernate Trigger to schedule the task to run on resume.

Answer 18

• An automatic unarchiver, because I got sick of remembering all the tar flags.

  The one I wrote was based of the file extension (look into os.path.splitext, or astring.split(".") for multiple extensions like tar.gz), but if that fails you could try and determine the file-type by calling the file command with subprocess

• Scripts to send IP updates to my no-ip and OpenDNS accounts (Mainly using urllib, and re to check the response)

• Tool to calculate total disc space, total used, total free, and display it in human readable terms. Could use either os module, or parse the output of the df command (using the subprocess and re modules). You also need to write a function to turn the number of bytes into GB/TB/etc

• Youtube FLV file downloader, perhaps using urllib to retrieve the source, BeautifulSoup to parse the HTML, and urllib again to download the file (the Firefox Live HTTP Headers addon helps working out how it resolves the flv URL)