Recap on past few days of downtime and problems

Wow... I'm sure many of you are wondering what the heck has been going on. ME TOO!


Here's basically the story:

We're running on a server that has a lot of different programs running on it. One of these programs is kinda like the operating system of your computer. The version we were on is really old, but has been working fine. It was suggested we upgrade to the newer version to take advantage of some nice improvements in performance. "The upgrade will only require about 30 mins of downtime." So we figured... "sure, why not." Well, what happened to us is kinda like getting notice to update your Firefox browser on your computer, but after you do, your computer completely melts down?

We try to schedule these things for the wee hours of the morning. That way as few peeps as possible are affected. The first night, we ran into a ton of problems during the attempted upgrade. We fixed a few things we thought would allow the upgrade to go through, but by that time it was 3:00 am PST (where I am) and peeps on the East coast were coming online, so we decided to wait a day.

The second day we tried again. The upgrade failed again, but for different reasons. We were working on fixing those problems, but again, East Coast started to come online.

Last night we figured all our ducks were in a row. The first part of the upgrade finally went through, and the second part seemed to be working, but then failed with a bang. We had to do a restore of a backup we made right before the upgrade (ALWAYS DO A BACKUP before you upgrade anything!!!) and while not upgraded, at least it seemed like everything was working again. Well, that was until we realized peeps couldn't send PMs So it was back to work, but this time it was more tricky because so many people were online. Imagine trying to do work on a 10 lane freeway in the middle of rush hour. Not fun or easy... sometimes impossible.

The bad news is, after all that work, problems, and downtime for the different parts of 3 days, we're still not on the newer software.

The good news: The backup system on BYC has been tested again and again and is solid, and while we had some downtime we believe we haven't lost any data!

This is kinda the nature of the beast (a cute and wonderfully friendly community beast) when running a huge site with a shoestring budget and small team.

Thanks everyone for your patience as we work through this stuff.

 

I hate Springing forward.