Revision as of 19:18, 17 August 2009

We have many shellservers, and we need a way to make sure they stay up to date as well as a system to automatically set up new ones. We have a custom set of scripts that takes care of this for us.

Features

• On any boot, will check the system integrity and update new files as necessary
• Can easily handle the 500000+files we have.
• Use our current cfengine setup to configure machines
• Be netbootable
• Requires no kernel patches
• Setup a new machine with no interaction

Overview

The system uses a list of precomputed md5sums as well as a list of all directories and symlinks to check the system integrity.

• The machine netboots off of a standard kernel and initramfs. It starts up normally, but hits a few scripts that we have added.
• Before the system tries to mount the root filesystem, a script (local-top/make_partitions) will check for the given partitions and make them if necessary
• After mounting the root filesystem, the main script goes into action (init-bottom/ugcs_rsync). This script:
  1. Downloads the md5sum, directory, and symlink lists (from /var/lib/ugcs)
  2. Checks the directories, and removes "stale" directories and creates new ones (new ones are created by rsync'ing them off of the image server)
  3. Computes md5sums of every file on the system
  4. Diff's the md5sum list of this system with the reference system
  5. All changed files are first removed, then rsync'd back (this makes sure that files that shouldn't be there no longer exist)
  6. All symlinks are removed and then re-created through rsync
• After the system is booted into /sbin/init, we run some scripts in rcS.d to get the initial configuration (using cfengine).

Overview of debian initramfs boot process

When a debian system boots, it runs an init shell script from the initramfs (it is copied from /usr/share/initramfs-tools/init) This runs through several script directories to mount the root filesystem and run any initialization.

When the initramfs is created, a series of "hooks" are run to add additional content. These hooks can copy configuration files or executables over.

If you make changes to any of these scripts, you need to re-run update-initramfs to create the new initramfs, and then you need to somehow copy it over to the netboot server.

Implementation Notes

local-top/make_partitions

Partition names and sizes are hard-coded into this script. New LVM logical volumes are only created if they don't exist, so you can safely change these if you need to setup a new class of machines. Eventually, it might have some intelligence.

For some reason, mkfs.ext3 doesn't seem to work when it is copied into the initramfs. To overcome this, we have an init script in rcS.d make the ext3 partitions (currently just afscache)

init-bottom/ugcs_rsync

A simple way to do this would be to use rsync to copy everything. However, rsync has high overhead, even when most of the files are the same. To overcome this, we use our own md5sum lists and diff them. Also, customizing rsync to do exactly what we want can be difficult and error-prone. Find commands are much more robust, so our system uses them extensively.

init-bottom/ldconfig

Occasionally you will have an issue where the linker cache doesn't match up to what is on the system. To fix this, we run ldconfig just to make sure (it doesn't take very long)

hooks/make_partitions

This copies over the appropriate executables for the make_partitions initramfs script

hooks/ugcs_rsync

This copies over the appropriate files so ugcs_rsync has rsync, etc. It also copies over find and xargs since the busybox versions don't have all the features we need.