Recap on past few days of downtime and problems

Discussion in 'Announcements, Feedback, Issues, & Guides' started by Nifty-Chicken, Mar 12, 2011.

  1. Nifty-Chicken

    Nifty-Chicken Administrator

    Wow... I'm sure many of you are wondering what the heck has been going on. ME TOO! [​IMG] [​IMG] [​IMG]


    Here's basically the story:

    We're running on a server that has a lot of different programs running on it. One of these programs is kinda like the operating system of your computer. The version we were on is really old, but has been working fine. It was suggested we upgrade to the newer version to take advantage of some nice improvements in performance. "The upgrade will only require about 30 mins of downtime." So we figured... "sure, why not." Well, what happened to us is kinda like getting notice to update your Firefox browser on your computer, but after you do, your computer completely melts down?

    We try to schedule these things for the wee hours of the morning. That way as few peeps as possible are affected. The first night, we ran into a ton of problems during the attempted upgrade. We fixed a few things we thought would allow the upgrade to go through, but by that time it was 3:00 am PST (where I am) and peeps on the East coast were coming online, so we decided to wait a day.

    The second day we tried again. The upgrade failed again, but for different reasons. We were working on fixing those problems, but again, East Coast started to come online.

    Last night we figured all our ducks were in a row. The first part of the upgrade finally went through, and the second part seemed to be working, but then failed with a bang. We had to do a restore of a backup we made right before the upgrade (ALWAYS DO A BACKUP before you upgrade anything!!!) and while not upgraded, at least it seemed like everything was working again. Well, that was until we realized peeps couldn't send PMs [​IMG] So it was back to work, but this time it was more tricky because so many people were online. Imagine trying to do work on a 10 lane freeway in the middle of rush hour. Not fun or easy... sometimes impossible.

    The bad news is, after all that work, problems, and downtime for the different parts of 3 days, we're still not on the newer software.

    The good news: The backup system on BYC has been tested again and again and is solid, and while we had some downtime we believe we haven't lost any data!

    This is kinda the nature of the beast (a cute and wonderfully friendly community beast) when running a huge site with a shoestring budget and small team.

    Thanks everyone for your patience as we work through this stuff. [​IMG]
     
  2. DuckLady

    DuckLady Administrator

    33,778
    3,723
    571
    Jan 11, 2007
    NE Washington State
    [​IMG]
     
  3. eenie114

    eenie114 Completly Hopeless

    Ohmigoodness. [​IMG]

    Well, looks like you all handled it really well! [​IMG]

    Now get offline and GO TO BED! [​IMG] We can't have you falling over.
     
  4. Coopa Cabana

    Coopa Cabana My Coop Runneth Over . . .

    3,236
    52
    241
    Aug 30, 2009
    Colorado
    My Coop
    Aren't you glad you get to set the clocks ahead tonight and not back! [​IMG]
     
  5. 77horses

    77horses ◊The Spontaneous Pullet!◊

    7,506
    505
    386
    Aug 19, 2008
    Maine
    Wow, it's a good thing BYC has such a great team! [​IMG] Hopefully it all works out for the best in the end. [​IMG]

    Thanks Nifty and the rest of the team for all the work and effort you put forth in order to continue making BYC the best it can be! [​IMG] [​IMG]
     
  6. ranchhand

    ranchhand Rest in Peace 1956-2011

    13,295
    43
    291
    Aug 25, 2008
    SC
    Quote:Wouldn't that be losing an hour? An hour he could spend chasing gremlins or even sleeping? [​IMG]
     
  7. Coopa Cabana

    Coopa Cabana My Coop Runneth Over . . .

    3,236
    52
    241
    Aug 30, 2009
    Colorado
    My Coop
    Quote:Wouldn't that be losing an hour? An hour he could spend chasing gremlins or even sleeping? [​IMG]

    Yes, he may lose an hour of sleep, but if he set the clocks back, he might have to re-live the nightmare.
     
  8. Nifty-Chicken

    Nifty-Chicken Administrator

    I hate Springing forward. [​IMG]
     
  9. DuckLady

    DuckLady Administrator

    33,778
    3,723
    571
    Jan 11, 2007
    NE Washington State
    Quote:I relish springing forward in life and not falling back, but the time change thing doesn't float my boat.
     
  10. andalusn

    andalusn Songster

    916
    31
    181
    Sep 6, 2009
    Ridgefield, WA
    Working for a software company I know of what you speak. I am on the support side but we also run an install upgrade side. Lots of planning and timing go into major upgrades. Sometimes not as expected

    Congrats to you on handling a very stressful situation that I am sure is not completely done.
     

BackYard Chickens is proudly sponsored by: