More Unexpected Downtime
June 15th, 2010Sorry, folks, but the server crashed again late yesterday afternoon. I’m going to have it up and down for the next hour or so.
Sorry, folks, but the server crashed again late yesterday afternoon. I’m going to have it up and down for the next hour or so.
Apologies for the downtime today. After an almost seamless upgrade to Ubuntu 9.10 the server had two completely unexplained crashes within the first 90 minutes back online. I paid a visit to the colocation facility, but couldn’t find any issues or get the crash to repeat. I used my time there to clean out some old kernel cruft, so maybe that will help. Anyway, I’ll post a real maintenance window before I do the next upgrade.
As promised, I finally made new certificates for all okcomputer.org TLS and SSL-enabled services. The new certs were pushed into production for HTTPS, SMTP STARTTLS and IMAPS this afternoon, and (so far) they seem to be working fine.
As usual, my recommendation is to set-up the okcomputer.org CA certificate as a trusted authority in your web browser and/or email client, and you won’t be bothered by certificate warnings until 2013. (The new service certificates will expire in February 2011.)
Please contact me if you have any questions.
After over a year since our last fiasco, I have finally gotten around to restoring the okcomputer.org homepage. Rita, I hope you’re happy. Please let me know if you’re having any problems with your favorite okcomputer.org sites and services.
In the meantime, we’ve added a new affiliate site, kip-kids.com, for Keneseth Israel Preschool.
It also appears that all of our application certificates have expired. I’ll update them this weekend.
okcomputer.org will be upgraded from Ubuntu Linux 7.10 to 8.04 Friday morning, May 23, between 10:00AM and Noon EDT. Most services will experience some downtime during that time period. Please let me know if you have any questions.
Update (Fri May 23 2008, 11:46 AM): The server has been upgraded and most services appear to be running well. Service testing will continue throughout the day. Please let me know if you encounter any issues.
Shortly after the big server rebuild a few weeks back I was examining the web server logs and I noticed that several sites were stealing okcomputer.org bandwidth by linking images from the okcomputer.org Photo Gallery directly into their web sites. So while I was rebuilding the web server I implemented a very common Apache mod_rewrite recipe so that these bandwidth stealers would get a very different image than the one they linked to — nothing gross, just the pic of me playing the ukulele that I drag out every now and then.
Well, apologies if you accidentally got that photo when you were trying to print your Photo Gallery pictures through Shutterfly. I hadn’t thought about that, but it’s fixed now.
I will be upgrading all WordPress installations tonight from version 2.3.2 to 2.3.3. Blogs will be offline for a short period of time around Midnight.
In the interest of security, I have already updated the version of the vulnerable xmlrpc.php file to a patched version in each installation with no obvious ill effects.
I spent last night rebuilding the okcomputer.org certificate authority and creating new certificates for all TLS and SSL-enabled services. Very early this morning I pushed the new certificates for HTTPS, SMTP STARTTLS and IMAPS into production and they seem to be working fine.
The beauty of this is that you can now set-up the okcomputer.org CA certificate as a trusted authority in your web browser and/or email client and you won’t be bothered by certificate warnings for the next five years. (The individual service certificates expire yearly.)
I’ll try to get a page up here soon explaining how to import the CA certificate (you should just be able to click the link above to install it in your browser), but please feel free to contact me if you have any questions.
User accounts, IMAP service and SMTP will be migrated to the new server Friday night, January 11, between 10:00PM and Midnight EST. During this time period SSH logins will be disabled (including scp) and mail services will not be functional. Incoming mail will be queued on an external server so nothing should be dropped.
Please let me know if you have any questions.
UPDATE 1/12/2008:
I’ve synchronized all login accounts and moved SMTP and IMAP to the new server. IMAP and SMTP relaying have tested good using SquirrelMail and Thunderbird. Exim is receiving mail and testing it with SpamAssassin.
Jessica reported a password problem, which has me a bit nervous, so please try to test your accounts ASAP and let me know about any problems.
During yesterday’s SquirrelMail migration I re-remembered that you can’t use multiple SSL certificates for different Apache NameVirtualHosts. This has put the kibosh on my plan to split all the okcomputer.org web services out to different DNS hostnames — everything’s gotta move back under www.
So far that will probably only affect the Photo Gallery, the Kid Amnesiac blog and the new okcomputer.org home page (i.e. the new apps I’ve implemented since the rebuild). SquirrelMail went under www where it needed to be since I discovered the problem while migrating it yesterday.
The new structure will look something like this:
I’ll try to schedule the change for tonight. Please let me know if you have any questions.
UPDATE 1/10/2008:
Last night’s migrations ended up being a bit more complicated than I thought. After a bit of pain Gallery and the okcomputer.org home page have been migrated. The Kid Amnesiac blog will have to wait for tonight.
UPDATE 1/11/2008:
Last night’s Kid Amnesiac migration didn’t go as expected either — Jessica told me she’d break my face if I moved the blog URL again. So instead I just moved it off of the Debian wordpress package and onto a clean copy of 2.3.2 and left the URL intact. There was some database pain involving character codes from where blog entries had been copied in from Microsoft Word, but that appears to be fixed now. I also decided to move the home page out of /home and into the root — much easier to do know that I know how to separate out the index.php file from the rest of the WordPress installation.