“Bad things happen in 3s” …???Posted: February 18, 2012
Whoever said this, I hope they were right. Here’s a peek at the network woes that have plagued Hamden Hall in the past month.
1) Bad network hub caused a broadcast storm that took down our network. After a few different methods of troubleshooting, we were able to locate it the “old fashioned” way… we unplugged the troublesome ethernet cord from the port on the switch and waited until someone yelled. We replaced that darn little NetGear hiding in the Art Cottage. (Yes, I know we should know better … NetGears are famous for that!)
2) The following week, a mouse ate through the fiber that connects our servers to the rest of campus… Those darn little teeth! It was repaired and resolved. We were down for about a day and a half. (Email worked from home though, thank goodness.)
3) The last, and most challenging of all the issues: Active Directory DNS database corruption. This problem became noticeable between events 1 and 2. We attempted to repair and “band-aid” the database as much as possible until the band-aid couldn’t hold it any longer–this happened Tuesday (Happy Valentine’s Day) at about 2pm. We worked through the night (literally) into Wednesday morning to rebuild 3 core servers to ensure that the problem had been resolved without any band-aids!
Now– we have had some unique issues at Hamden Hall before. Worth noting is the pipe that burst in the server room on the first day of school three years ago– water shooting directly into our core switch and trickling into the remainder of the undeployed PCs. We put those wet PCs out behind the server room on a pallet to dry in the sun, only to be hit by a car coming around the turn of the drive way. Who can make this stuff up? But never have we had 3 unrelated issues hit us this close together to cause such an interruption in teaching, learning and productivity. (I will reflect a little more about this in a different post. )
The purpose of this post is to publicly thank all of those people who helped to get us to where we are today.
1) My Network Manager, Victoria Helland, who gave 110% trying to support and resolve all of these issues. She truly carried the burden of the network outages on her shoulders and she did everything humanly possible (and without much sleep) to get us up and running. I am lucky to have her on my team!
2) My Husband Jon, who not only was an amazing support to me emotionally through all of this, but was full of what seemed to be unlimited resources when it came to knowing the right people at the right time. I am lucky to be married to him and grateful for his vast experience in the IT field throughout his career. Not only did he hook us up with some great people, he called them to make sure they took good care of us. I also thank him for handling all the drop offs, pick ups, homework, cooking, and for packing me everything I needed to start the day on Wednesday (after the all nighter). The outfit matched (I was a little worried). Thanks for checking on me all through the night, listening to me, asking the right questions to help me troubleshoot, and being a shoulder for me to cry on when I finally lost it. Sorry for missing Valentines Day. I love you.
3) My kids, who listened to my endless conference calls on speaker phone on the way to and from school and who were able to “go with the flow” staying late at school, and coming in early. They are strong and independent (YAY!) and I am very proud of them!
4) Jim Hunter, Director of Facilities, who always goes above and beyond to support the tech department. From coming in at 5am to help the fiber technician pull the new fiber in the dirt basement to delivering endless food, coffee and words of encouragement. He truly epitomizes the Hamden Hall spirit of commitment and dedication. We couldn’t have gotten where we are today without his help. His consistent “bright side” thinking helped us all plow through the issues.
5) Chris Buckley and Sarah Ludwig, who were on the front line supporting teachers and students through all of the ups and downs. You did this with a smile and patience. I am lucky to have such a great team!
6) FiberOptics Plus (a referral from my husband Jon), and especially the tech Matt, for quickly getting to campus to figure out where the break in the fiber was, and for coming back to campus at 5am the morning after the SuperBowl (Go Giants!) to get us working by 9:30am (3 hours ahead of the original projection).
7) Brian Diamond, President and CEO of LANStatus: ”Exceptional Attention. Extraordinary Results”– very true! (also a connection from Jon) Thanks, Brian, for sending us our savior, Nate Ravid. Nate showed up a half hour ahead of schedule this past Tuesday, and stayed with us until 4AM on Wednesday. He was extremely knowledgeable, calm, and got the job done! We successfully restored from backups, rebuilt our Active Directory and two domain controllers, a print server, and manually rejoined more than 50 computers to the domain. ALL before 8am when school started the following day. On Wednesday (February 15th), every class went on as scheduled with the technology they needed. It was my mission to make that happen and WE DID IT! Pretty incredible if you ask me! Thanks again Nate. It was a pleasure having Modern Pizza and cookies with you at midnight. I apologize if you didn’t spend Valentines day with a special someone because of it.
8) Bob Cashman of CCA and his consultant genius Dana Urban, who answered my dire call for help at 6pm on Wednesday (after our all nighter). Our email server was the last of the casualties and Victoria and I were brain dead at this point. These two men spent 4 hours on the phone helping us trouble shoot our issues and get a plan together for the morning. Their help, along with First Class Support, guided us to the best case scenario possible on Thursday morning. We had our email server up and running by 11am.
9) TechnologyPartners, our online backup company who answered countless phone calls helping us restore all servers with minimal data loss.
10) Andrew Speyer, Director of IT at Choate, one of my CAIS CoT best buds, who answered my “HELP!” text within minutes when I needed a contact for a fiber technician.
11) The many patient students and faculty of HH, whose endless cheers and words of encouragement and appreciation kept us going. Thanks for not “giving up” on technology and for realizing that sometimes these things can happen. (Heck if some of the big companies (Google, Twitter, Facebook), with unlimited funds can have network issues, we’ve done pretty well over the years!)
12) And last (but not least), my Head of School, Bob Izzo, who was also encouraging, patient, trusting and actually joking with me through all of this. I appreciate his support and I feel lucky to have him as my boss. He could have been A LOT less patient. And I thank him, in advance, for listening and supporting our proposal for upgrades to our infrastructure
So, if you stuck with me through all of these thank yous, I ask that you follow these amazing people on Twitter or LinkedIn, support their businesses, read their blogs, etc. They are an amazing bunch and I am eternally grateful. I hope I didn’t leave anyone out. If I did, please forgive me. I am still catching up on my sleep! It was a wild few weeks (especially this one). I truly hope that bad things happen in 3s. It would be nice if the rest of the year was smooth sailing!