What methods/techniques/tools do you recommend for documenting systems and infrastructure (runbooks)?

Question

Background

I recently joined a small start-up after working at a large company.

I'm not a professional system administrator, however, because of my programming and systems knowledge I am now the internal person managing our servers and infrastructure.

In the past, I never needed to document our system information: passwords (for servers, databases, routers, switches, etc), which servers were running which applications (both homemade and installed), server IP addresses, configuration file locations, etc... The professional system admins always did this work, freeing me to focus on other areas.

Event that triggered urgency

I'd been a bit casual about moving this forward until I discovered that I didn't know where my main subversion configuration file was in /etc/apache2 (not to mention that the config file isn't under version control or backed up!) I realized that this needs to be addressed quickly.

Next step

I now have to figure out how to document all of this in a sane, elegant, access controlled manner.

I've heard of runbooks, but I don't know the best way or tools to manage them. My first thought was an excel/openoffice spreadsheet under version control.

Is there a good guide to maintaining runbooks? Good software? This must be a fairly common problem, how do you handle it?

Answer 1

I've actually had good success with a Wiki. Use something where you can control logins easily — Mediawiki is okay, but requires some PHP hacking — and build some templates for processes, inventories, and so forth.

Update

Actually, I must have needed coffee. Trac is pretty nearly ideal; better access control, integrated issue tracking, and a somewhat stronger text model. You can even tie it directly to your subversion repository so that you can hook actual scripts to their runbook pages.

Answer 2

Wikis are indeed a good approach. You can set up sharepoint for that and get nice features like update history, links that always point to current information and such.

Answer 3

If you use something like Puppet, (or Chef, or CFEngine, etc.) to build and manage your machines, then most of the runbook contents can live inside the puppet config, leaving you with a lot less stuff (just the location and password for the puppetmaster, if you take it to extremes!) to put in your wiki.