System Administration Best Practices: Documentation

System Administration Best Practices: Documentation

There are tons of best practices out there, but it helps to keep
things to a theme. Documents are our theme this time. I’ve got
other plans for this section next week, but this can be one of
those stop-and-start series if people are interested.

Documentation comes in two forms here: policies and history. You
document policies as far as those little details, like onto which
directory you mount the project files on all of the machines
(hopefully the same one each time), and from which machine those
project files are mounted, using what protocol (NFS? SMB?).

Policies can also include the terms of use for your staff. What
are they allowed to access? What aren’t they? How is it monitored
and policed? What are the penalties for breaking the rules? How do
you make sure the staff is aware of the rules? (Very
important!) In this case, you have to have discussed these with
your management and maybe even a legal expert, and then you need
to make sure everyone signs off on having read the policy, and
repeat this whenever the policy is updated.

History is equally important, but for a different reason. One day,
you might be on vacation, and suddenly you get an urgent phone
call from work because no one can figure out what the root
password is for a particular machine, and they desperately need
access to get things up and running.

Of course, the danger of putting passwords down on paper or in
electronic files is that someone unauthorized might find them. Put
the master passwords list under lock and key in the server room
with only the top level administrators having access. Or, if you
put them under a combination lock, if you have to you can walk a
junior admin or your spouse through opening the safe, logging in,
and so on.

If you have to put things like this down on paper, be sure to not
put, “Here are our most vital passwords” on the front of the
notebook in hot pink paint! Figure out a way that the people who
need to find them know what to look for, but it’s not obvious to
the casual observer.

You also have the option of using some kind of password management
tool. There’s a lot of them out there on the Web. Some people just
use a PGP-encrypted file to store all of their passwords, so they
only have to memorize one keyphrase and the rest they can look up
when needed.

Robert
LeBlanc shares one of his documentation strategies here. He’s
both a system administrator and programmer, and tends to build
everything from source, so this is of course biased toward such an
approach:

“[This] is the sort of thing that every system
administrator should be documenting for himself somewhere, for
every package installed on his system. It’s another kind of
‘backup,’ in case you have to rebuild your system at some point in
the future, and it enables other system administrators who may
inherit your work to understand how you’ve configured everything.
Most organizations of any size should be requiring their
administrators to maintain such documents (sometimes called the
’system bible’ or somesuch) for that very purpose, and an up-to-date printed version should be stored along with your backups (on-
site and off-site). It’s just good policy, even if you’re a one-person shop - one day, you may have to re-install or upgrade
something, and you’ll be thankful you documented all the little
‘trouble spots’ the first time around.

“My own system documents tend to average around 100 pages or so,
due to the fact that I also include the complete contents of every
configuration file. At the beginning of the document I provide an
overview of the system - the hardware specs, OS, library versions,
build environment, the partition table, and a list of the IP
interfaces and domain names the machine services. After that, I
list the installed packages in alphabetical order, with the
following information about each one:

• Installed version
• Installation date (date this version was installed)
• Installed by (the admin who installed this version)
• Purpose (one-sentence description of the package)
• Package URL (the software developer’s website)
• Download URL (where to get the latest version)
• Configure options (list of flags for ./configure or make, if
  applicable)

• Application path (where the application is installed)
• Configuration path (where the config files are installed)
• Logs (where the application writes its logs)
• Data (where the application’s data is stored)
• Serial number (registration key, if any)
• Prerequisites (any other packages that must be installed
  first)

• Modifications (any patches applied to the package)
• Notes (any additional notes regarding this package)
• Configuration files (full contents of all of the application’s
  config files)

“This is just the good practice of documenting your work,
especially if you maintain multiple systems - keeping track of
what exactly is installed on which system can lead to serious
headaches without some sort of documentation. :) The next time you
have to build something like PHP from sources, you’ll be thankful
you kept a list of the 20+ configure options you used last year,
believe me. It’s also nice to be able to easily find out where the
documentation for the package can be found, where to check for and
download the latest version, and where on your system you actually
installed the various components of an application.”