Splunk runs a bunch of saved searches that can activate alerts. Log in to splunk and go to "admin" and then "saved searches". Splunk saved search scripts are located at charon:/opt/splunk/bin/scripts

Users logging into coreservers

This saved search runs once a minute and scans between one and two minutes ago for invalid users logging into coreservers. It then runs a script (muggles_trying_coreservers.py) that sends the user a helpful email. It has a couple of protections to protect people from getting notices about ssh brute-force attempts.

LDAP server down

This alert sends sysadmins and sysadmins non-ugcs addresses notices if there are too many "ldap server down" messages. If you get it, double-check that Hera and Zeus are up and running correctly

Mail forwarded in past 15 min

This alert checks to see if we've forwarded any mail in the past 15min. If we haven't for a few periods, it is a likely indication of problems. It emails sysadmins and external sysadmins if it finds a problem. If you get one in the middle of the night, it's not a big deal. If you get 3 or 4 in a row, look through postfix logs for errors.

Client key expired

This alert lets you know if a Kerberos principal has expired. If one has, you should go reset its expiration date. This is especially important for server principals.