Revision as of 07:29, 4 November 2007

News

Current

Archived

November 1, 2007

We are planning some downtime for the afternoon and evening of Saturday November 3 in order to install gigabit ethernet cards in our mass fileservers and bring additional backup servers online. There may be disruptions to your ability to access files in your home directory, although we will try to minimize impact as much as possible.

Update: the downtime should be largely finished, except for rolling reboots on our shellservers. Apologies for the disruption to file and e-mail services, a few things didn't go as planned, but everything is back up now, and performance should be vastly enhanced. --Elizabeth@ugcs.caltech.edu 21:32, 3 November 2007 (PDT)

October 25, 2007

2:20am: We appear to be under an smtp DDoS attack. We are taking measures to mitigate this attack. To reduce ldap load, our ldap database has been set readonly, so any changes to user information will not work. Also as a preventative measure, we have added more restrictive filtering rules for our core servers- including blocking non-Caltech Kerberos requests (this will be unblocked in the morning). If you notice that we have blocked a critical service, please contact us. We will post updates here as we have them Jdhutchin@ugcs.caltech.edu 02:25, 25 October 2007 (PDT)

Update All services should be back to normal Jdhutchin@ugcs.caltech.edu 11:13, 25 October 2007 (PDT)

October 24, 2007

We recently needed to restart a large number of services due to the openssl 0.9.8g update. If you notice any problems, please contact us. Additionally, we experienced 3 hours of website homedir downtime on the afternoon of the 23rd due to an AFS readonly mirror bug. We have disabled the readonly mirror behavior until we can investigate and things appear to be stable for now.

October 15, 2007

We have recently fixed the issue with Apache directory listings and believe that we have something very similar now up and running. The cluster is once again at full function, following unplanned downtime that occurred on October 13 due to power disruption.

October 13, 2007

We lost power and network to the cluster this morning due to some miscommunication with physical plant about the extent to which a power maintenance event would impact us. We are still scrambling to get everything up and working again, fingers crossed Update Services are up and running again. We have also added more shell servers to the to.ugcs rotation.

October 11, 2007

We will be doing a considerable amount of work on the cluster on Saturday, October 13. We do not expect any major downtimes; however, some login servers will be taken offline for short periods of time, and core servers may be rebooted causing brief interruptions. If you notice that something has been down for more than 5 minutes on Saturday, please contact us

October 3, 2007

All home directories were transferred as of Monday, October 1. Some files may have incorrect ownership since the copy was performed as root. If you have any files that were not transferred, or if you need to access the backups of UGCS3 files, please contact the sysadmins and we will copy your files from backup.

September 30, 2007

Home directories are now being copied into place; the process is about 30% complete but will run automatically (although a few users ran out of quota while I was copying and I'll need to reset their quotas and retry the transfer of their homedirs). There will be duplication of a lot of files - in particular, I've preserved your mail and www and cgi trees despite their having already been copied once before to a different location. To decrease your quota usage, you should delete any files you do not need immediately. Thanks for your patience!

September 28, 2007

All UGCS3 services have been shut down and transferred to UGCS4 servers, which are all now in the racks in the server closet. Home directories have been archived and will be shortly copied to the new cluster home directories. We do not anticipate any further major downtime.

If you still need to migrate your password, please use the link to the left. If you've forgotten your password, please e-mail sysadmins@ugcs.caltech.edu - if you used SSH key login previously, we will put up your SSH key on a machine that will use your key to authenticate you and allow you to set a password on the new cluster.

Thanks very much for your patience, and we look forward to serving you.

Your UGCS Sysadmins

September 26, 2007

Home directories on UGCS3 will be frozen in the evening of September 27 in preparation for mass copying to the new fileservers. We are planning to bring down all services beginning as early as the afternoon of September 28 in order to rackmount all of the new hardware, carry out IP address reallocations, and reimage old core servers as user-accessible shell servers in UGCS4. The migration should be fully complete by the end of Saturday September 29.

September 22, 2007

Please make sure to read the Migration FAQs as they answer a number of topics related to the impact on user services during the course of the migration.

September 13, 2007

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear UGCS users,


We are proud to announce that we are about to complete the rollout of
UGCS 4.0, which will offer improved performance, features, and quotas.
However, these changes will require some action on your part, as well as
awareness that many of the quirks of UGCS behavior you are used to will
no longer be present.  If you have any questions, comments, or concerns,
please leave us an e-mail at sysadmins@ugcs.caltech.edu and we'll
respond as soon as we can.

===Bottom line===
* We are targeting Saturday September 29 for the main switchover.  All
services will be unavailable on that day.
* Your password is frozen in its current state as of September 12.  You
will need to log into https://hermes.ugcs.caltech.edu/password.html with
it to access mail, and to use the rest of the cluster when the migration
is complete.
* Your mail will only be accessible via secure IMAP and POP3, effective
very soon (tentatively September 16); you will need to verify your
password first as stated above.
* /ug/drop/mail is no longer writable, and all existent mailing lists
(except one-member lists) will be transferred as they are in their state
as of September 12.
* SSH keys will no longer work after the migration; we recommend use of
Kerberos for passwordless authentication.
* After the migration, your home directory will be copied to your new
home directory.  You will not be able to set per-file permissions, only
per-directory permissions.  The main portion of your home directory will
be not readable by anyone other than yourself; in order to share files
with other users, you will need to place them in the public subdirectory
of your home folder.
* If you wish to help us beta test the new system please send us an
e-mail and we will provide login instructions for our test machines.

===Authentication===
We are migrating from NIS, which stores crypt() passwords, to Kerberos;
since crypt() is irreversible and Kerberos requires a copy of your
secret to create your principal, we cannot directly perform this
migration for you.  You will need to enter your old password and a new
password into an online form (using SSL). The application will then
enable your kerberos principal which you can subsequently use to access
all services on the cluster after the migration is done. Your migrated
password will be usable with mail (IMAP/POP3) immediately.  The
migration URL is the following:
https://hermes.ugcs.caltech.edu/password.html
The SHA1 fingerprint of the temporary self-signed certificate (until we
have time to properly establish a CA) is
22:44:7D:F3:D9:44:A0:59:CA:B4:AC:70:5A:F5:94:9A:3F:2C:4F:15

===Network Filesystem===
We are migrating from NFS to AFS, a filesystem in wide use at other
universities including Stanford, MIT, and Carnegie Mellon.  AFS has
vastly improved security and speed compared to the version of NFS
currently in use on the cluster, not to mention better administrative
tools which will allow us to easily back up your data and move it
between servers to maximize performance.  AFS also allows user-settable
ACL's, eliminating the need to create custom groups for allowing subsets
of users access to data.  However, there are a few caveats: AFS does not
store permissions by file, only by directory.  We are defaulting to have
home directories remain readable only by their owners, with a
world-readable public subfolder.  If you wish to add a public file to
your home directory, place it in the public folder and symlink the
filename in your private home directory to the equivalent in your public
folder.  We have already set up a few such commonly-used symlinks on
your behalf such as .plan.  We will migrate your data for you from NFS
and place it in your home directory during the migration.

We have acquired approximately 3.2 terabytes of mass storage and 0.3
terabytes of fast SAS storage.  As a consequence, we are setting initial
quotas to 500 MB of mass storage for your home directory and 150 MB of
fast storage for your mail.  We reserve the right to modify these quotas
in the future, although they will most likely rise.  If you wish to have
a larger mail quota, please contact us - we can move your mail spool to
one of the mass storage machines and give you more space (at the penalty
of performance).

===SSH===
Your SSH keys will no longer function.  This is deliberate - AFS uses
Kerberos for authentication, which means that a Kerberos ticket is
required to mount your home directory; SSH keys cannot not provide
Kerberos authentication.  If you SSH to a machine directly and enter
your password, Kerberos tickets and AFS tokens are automatically
obtained for you using your password.  If you wish to use passwordless
authentication, we recommend that you install a Kerberos client on your
system and enable forwarding of tickets over SSH (GSSAPIAuthentication
and GSSAPIDelegateCredentials) for *.ugcs.caltech.edu in your
.ssh/config file if using *nix.

We are in the process of acquiring a number of new user-accessible Core
2 Duo systems, but all of the puke-class Pentium III machines will be
migrated for the present and the servers used for UGCS 3.0 services will
be decommissioned over time and integrated into the cluster as
user-accessible shell systems.

===Mail===
We have switched to using Maildir format for delivery of all new
messages.  Maildirs perform significantly better in a network filesystem
environment by avoiding the need to lock a single mbox file.  IMAP and
POP will only show messages from your Maildir.  We have used mb2md
(http://batleth.sapienti-sat.org/projects/mb2md/) to place all the
messages from the mboxes we could identify in your Maildir.  If you wish
to manually migrate additional mboxes after the migration, you can
invoke mb2md yourself.  All inbound e-mail is now filtered using
amavisd, spamassassin, and clamav.  If you wish to forward your mail to
another address, you should update your LDAP entry with one (or
multiple) mailForwardingAddress entries instead of relying on .forward.
 Procmail is currently not in the mail delivery chain, but will be
integrated at a later date if it is still required by a large number of
users; we anticipate that the new mail stack will suffice for the
majority of users that were using procmail to invoke spamassassin or
perform filtering.  Additionally, since we are now able to filter all
inbound mail, we no longer need to greylist e-mails and therefore you
will no longer experience delays in delivery of mail to ugcs addresses.
 We have disabled non-secure IMAP and POP; you will need to use IMAP/S
or POP/S instead.  Like SSH, our IMAP and POP services are Kerberized
and you can authenticate without entering a password if you have a
Ticket-Granting-Ticket.  If you wish to send outbound e-mail using
UGCS's SMTP server, you also will need to authenticate either using your
password or a Kerberos ticket.

===Webmail===
We are offering two new options for accessing your e-mail from a web
browser.  Roundcube is an AJAX webmail client that behaves like a
desktop mail client with drag and drop support.  Squirrelmail is more
traditional and works for the more paranoid about Javascript.  You can
go to https://hermes.ugcs.caltech.edu/roundcube or
https://hermes.ugcs.caltech.edu/squirrelmail to access them.

===Mailing Lists===
We will be migrating all /ug/drop/mail lists to Mailman, a widely used
mailing list management tool that offers additional features such as
automatic removal of spammy messages, blocking of posts from
non-members, moderation, unsubscription, and archiving of messages.
Existing /ug/drop/mail lists have been frozen in preparation for the
migration.  We are offering a web-based list administration tool located
at https://hermes.ugcs.caltech.edu in place of /ug/drop/mail.  For those
who use automated tools to manage /ug/drop/mail lists, please contact us
and we will advise you of the best way to handle automatic
additions/removals of list members.

===Public webhosting===
Your public_html folder will be automatically migrated and be served
from our new webserver.  We support PHP (version 5) and Perl through
SuExec.  By default, the web server will not be able to read files from
your home directory - if your website relies on files outside of the
public_html directory, they should by symlinked or moved into
~/public/public_html/. If you have questions about migrating your
existing web applications, please contact us.

===Software===
Cluster machines will be running Debian testing (Lenny) with a set of
commonly used packages.  If you'd like to request a piece of software
which is currently not installed, please contact us and we'll add it to
the standard system image.  We hope that this central package management
will allow us to keep the software on UGCS as up-to-date as possible
with new versions and security updates.

===Database services===
Currently, database migration is not automated.  Please contact us to
get your database created and/or migrated.

===Chat===
In addition to continuing to support Gale, we are planning to set up a
Jabber server for your chatting convenience.

===Hosting and Authentication===
As the result of rearrangements made to our very limited pool of 62
usable IP addresses, we have needed to change the block of addresses
allocated to third-party hosting.  If you are hosting a server with us
and we have your contact information, we will send you your new
information to place in /etc/network/interfaces and will expect you to
configure your server appropriately or provide us with the access to
change the IP ourselves.  If not, you will have to track us down when
your server stops working.  In particular, there are a few bits you
_need_ to pay attention to with respect to specifying the correct MTU,
netmask, and routes.  Also, if your server remains offline for more than
a period of two weeks and we have no contact information on file for
you, we reserve the right to reallocate your IP to someone else.  If
your server is currently offline, we cannot automatically gather its MAC
address and will need this information from you if you wish to have an
allocation in the new network scheme.

As always, individuals in the Caltech community are welcome to colocate
servers with us.  We ask that you provide us with current contact
information in event we need to disrupt service to your server; we also
require your server's MAC address in order to place it on the
appropriate VLAN and provision you with a static IP.  We run network
intrusion detection software (Snort) to protect your server and also can
tighten firewall rules to restrict inbound traffic if you so desire.

Our Kerberos infrastructure is also available to others operating web or
other applications who need to validate the identity of a member of the
Caltech community.  Contact us for details if you are interested.


Regards,

Your UGCS sysadmins (Elizabeth Fong, Matthew Maurer, Joshua Hutchins,
and Alex Roper)
sysadmins@ugcs.caltech.edu
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG6QL8aj+LAPvd0qQRAuR4AJ0R7g4+EBAreYV9qPENRMTpsIpEhACdHgpn
hUKMufALmxRFTL2JpYpZGJk=
=Pv2m
-----END PGP SIGNATURE-----

August 22, 2007

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dear UGCS users,


As the 20-year anniversary of UGCS in 2009 approaches, we are making
preparations for the next 20 years of UGCS in order to ensure that the
cluster is used by as many people as possible and continues to provide
top-notch services to the Caltech community. We are proud to announce
that we have been planning significant hardware and software upgrades to
UGCS over the course of the past six months which will result in vastly
improved performance, features, and quotas. In short, we are moving all
UGCS services to new, faster hardware and retooling the software
architecture to use commodity, well-supported software that we can
update and maintain in the coming years.

We hope to be finished with the initial migration by the beginning of
October. Please be advised that some UGCS services may need to be
temporarily disrupted during the buildout. Additionally, we may snapshot
the /ug/drop/mail system and the user password database for migration;
any changes following the snapshot will need to be reapplied after the
migration. A week before the migration, we will advise you of what
changes will impact you and any actions you may need to take. When we
switch over to the new infrastructure, we will need to bring down all
UGCS services for approximately one day.

If you have any questions, comments, or concerns, please send us an
e-mail at sysadmins@ugcs.caltech.edu and we'll respond as soon as we can.


Sincerely,

Your sysadmins (Elizabeth Fong, Matthew Maurer, Joshua Hutchins, and
Alex Roper)
sysadmins@ugcs.caltech.edu
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGzOZ4aj+LAPvd0qQRAgANAJ0RPhpm5+Lwwk10ItEZdEivgx/UlwCggVnn
TaPLKRdEjt2yGcnDH7ygtw4=
=qoiU
-----END PGP SIGNATURE-----