Recap on past few days of downtime and problems

Nifty-Chicken

Administrator
BYC Staff
14 Years
Dec 26, 2006
41,590
34,367
1,552
California - SF East Bay
My Coop
My Coop
Wow... I'm sure many of you are wondering what the heck has been going on. ME TOO!
barnie.gif
he.gif
wink.png



Here's basically the story:

We're running on a server that has a lot of different programs running on it. One of these programs is kinda like the operating system of your computer. The version we were on is really old, but has been working fine. It was suggested we upgrade to the newer version to take advantage of some nice improvements in performance. "The upgrade will only require about 30 mins of downtime." So we figured... "sure, why not." Well, what happened to us is kinda like getting notice to update your Firefox browser on your computer, but after you do, your computer completely melts down?

We try to schedule these things for the wee hours of the morning. That way as few peeps as possible are affected. The first night, we ran into a ton of problems during the attempted upgrade. We fixed a few things we thought would allow the upgrade to go through, but by that time it was 3:00 am PST (where I am) and peeps on the East coast were coming online, so we decided to wait a day.

The second day we tried again. The upgrade failed again, but for different reasons. We were working on fixing those problems, but again, East Coast started to come online.

Last night we figured all our ducks were in a row. The first part of the upgrade finally went through, and the second part seemed to be working, but then failed with a bang. We had to do a restore of a backup we made right before the upgrade (ALWAYS DO A BACKUP before you upgrade anything!!!) and while not upgraded, at least it seemed like everything was working again. Well, that was until we realized peeps couldn't send PMs
sad.png
So it was back to work, but this time it was more tricky because so many people were online. Imagine trying to do work on a 10 lane freeway in the middle of rush hour. Not fun or easy... sometimes impossible.

The bad news is, after all that work, problems, and downtime for the different parts of 3 days, we're still not on the newer software.

The good news: The backup system on BYC has been tested again and again and is solid, and while we had some downtime we believe we haven't lost any data!

This is kinda the nature of the beast (a cute and wonderfully friendly community beast) when running a huge site with a shoestring budget and small team.

Thanks everyone for your patience as we work through this stuff.
hugs.gif
 

77horses

◊The Spontaneous Pullet!◊
12 Years
Aug 19, 2008
7,590
613
476
Maine
Wow, it's a good thing BYC has such a great team!
th.gif
Hopefully it all works out for the best in the end.
fl.gif


Thanks Nifty and the rest of the team for all the work and effort you put forth in order to continue making BYC the best it can be!
clap.gif
thumbsup.gif
 

ranchhand

Rest in Peace 1956-2011
11 Years
Aug 25, 2008
13,295
84
291
SC
Quote:
Wouldn't that be losing an hour? An hour he could spend chasing gremlins or even sleeping?
big_smile.png
 

Coopa Cabana

My Coop Runneth Over . . .
10 Years
Aug 30, 2009
3,236
93
261
Colorado
My Coop
My Coop
Quote:
Wouldn't that be losing an hour? An hour he could spend chasing gremlins or even sleeping?
big_smile.png


Yes, he may lose an hour of sleep, but if he set the clocks back, he might have to re-live the nightmare.
 

andalusn

Songster
10 Years
Sep 6, 2009
919
46
191
Ridgefield, WA
Working for a software company I know of what you speak. I am on the support side but we also run an install upgrade side. Lots of planning and timing go into major upgrades. Sometimes not as expected

Congrats to you on handling a very stressful situation that I am sure is not completely done.
 

New posts New threads Active threads

Top Bottom