UGCS Best Practices

From UGCS

This page aims to document all the things we typically do so that new admins can get up to speed faster.

Machine Setup

Use LVM for all machines. If you don't, you're a moron.
Leave at least 128mb for a separate /boot partition. Otherwise, it may fill up and just be a pain
Leave some spare space on LVM- you don't have to allocate all your space to begin with.
Use xfs, reiserfs, or jfs where possible- these filesystems are more flexible than ext3, and can easily grow when necessary

remctl scripts go in /usr/local/lib/remctl unless they are part of a debian package
Read http://www.pathname.com/fhs/pub/fhs-2.3.html If you don't follow FHS standards and flagrantly violate them, you will confused everyone and should commit seppuku.
If you have a lot of stuff that doesn't need to be backed up, create a new afs volume for it so you don't eat backup space.

Any configuration file that is on more than one machine should be in cfengine. This way everything can easily be kept consistent. If you need to edit a config file, be sure to check cfengine.

If you solve a problem, try to set something up so that we will be automatically notified if the problem happens again. This can involve either a Nagios alert or a splunk saved search run on a schedule.

When setting up a new service, try to make it so that CFengine can take care of its configuration files. This way, it can be moved to other machines as necessary. To do this, make a new .conf file demeter:/srv/cfengine/global/inputs/, and then add a new class in cfagent.conf for your service so it doesn't get installed on every machine.
If possible, add tests for it in Nagios so that we'll be alerted if it fails. Try to test every part of your service (we have stuff that tests web, as well as php and cgi scripts, and everything again on a vhost so we know it all works)
Document new services!
If you need a config file with stuff about lots of different machines, use Configurator and add your script to update-cfengine on demeter (you did use cfengine, right?) This make stuff easier to automatically update, and it is easier for someone else to see what is going on, and stuff doesn't get randomly left out.

Avoid suid- with AFS, it's kinda useless (you'd need a keytab). Instead, use Remctl
You need to check input for several different things:
- Anytime you open a file, make sure the user can't play with your path by adding "../". A good idea is to run input (especially usernames) through python's 'isalnum()'
- Anytime you write to a file as root, make sure it isn't a symlink attack (usually you can just check to see if the file is a symlink)
- Anytime you run a command, user input needs to be escaped correctly. Since escaping stuff for shell is kinda hard, you should apply strict tests and re-build parts. For example, if they gave an email address, make sure it is only '[a-zA-Z0-9.@]' There are so many things you can do to confuse shells, so be very strict. Much better safe than sorry, especially when you're running as root.
- Anytime you run sql. We don't do much of this.