“Bad things happen in 3s” …???

Whoever said this, I hope they were right. Here’s a peek at the network woes that have plagued Hamden Hall in the past month.

The mice at Hamden Hall must have needed some more fiber in their diet!

1)  Bad network hub caused a broadcast storm that took down our network. After a few different methods of troubleshooting, we were able to locate it the “old fashioned” way… we unplugged the troublesome ethernet cord from the port on the switch and waited until someone yelled. We  replaced that darn little NetGear hiding in the Art Cottage.  (Yes, I know we should know better … NetGears are famous for that!)

2) The following week, a mouse ate through the fiber that connects our servers to the rest of campus… Those darn little teeth! It was repaired and resolved. We were down for about a day and a half. (Email worked from home though, thank goodness.)

3) The last, and most challenging of all the issues: Active Directory DNS database corruption. This problem became noticeable between events 1 and 2. We attempted to repair and “band-aid” the database as much as possible until the band-aid couldn’t hold it any longer–this happened Tuesday (Happy Valentine’s Day) at about 2pm. We worked through the night (literally) into Wednesday morning to rebuild 3 core servers to ensure that the problem had been resolved without any band-aids!

Now– we have had some unique issues at Hamden Hall before. Worth noting is the  pipe that burst in the server room on the first day of school three years ago– water shooting directly into our core switch and trickling into the remainder of  the undeployed PCs. We put those wet PCs out behind the server room on a pallet to dry in the sun, only to be hit by a car coming around the turn of the drive way.  Who can make this stuff up? But never have we had 3 unrelated issues hit us this close together to cause such an interruption in teaching, learning and productivity. (I will reflect a little more about this in a different post. )

The purpose of this post is to publicly thank all of those people who helped to get us to where we are today.

1) My Network Manager, Victoria Helland, who gave 110%  trying to support and resolve all of these issues. She truly carried the burden of the network outages on her shoulders and she did everything humanly possible (and without much sleep) to get us up and running. I am lucky to have her on my team!

2) My Husband Jon, who not only was an amazing support to me emotionally through all of this, but was full of what seemed to be unlimited resources when it came to knowing the right people at the right time. I am lucky to be married to him and grateful for his vast experience in the IT field throughout his career.  Not only did he hook us up with some great people, he called them to make sure they took good care of us. I also thank him for handling all the drop offs, pick ups, homework, cooking, and for packing me everything I needed to start the day on Wednesday (after the all nighter). The outfit matched (I was a little worried). Thanks for checking on me all through the night, listening to me, asking the right questions to help me troubleshoot, and being a shoulder for me to cry on when I finally lost it. Sorry for missing Valentines Day. I love you.

3) My kids, who listened to my endless conference calls on speaker phone on the way to and from school and who were able to “go with the flow” staying late at school, and coming in early. They are strong and independent (YAY!) and I am very proud of them!

4) Jim Hunter, Director of Facilities, who always goes above and beyond to support the tech department. From coming in at 5am to help the fiber technician pull the new fiber in the dirt basement to delivering endless food, coffee and words of encouragement. He truly epitomizes the Hamden Hall spirit of commitment and dedication. We couldn’t have gotten where we are today without his help. His consistent “bright side” thinking helped us all plow through the issues.

5) Chris Buckley and Sarah Ludwig, who were on the front line supporting teachers and students through all of the ups and downs. You did this with a smile and patience. I am lucky to have such a great team!

6) FiberOptics Plus (a referral from my husband Jon), and especially the tech Matt,  for quickly getting to campus to figure out where the break in the fiber was, and for coming back to campus at 5am the morning after the SuperBowl (Go Giants!) to get us working by 9:30am (3 hours ahead of the original projection).

7) Brian Diamond, President and CEO of LANStatus: “Exceptional Attention. Extraordinary Results”– very true! (also a connection from Jon)  Thanks, Brian, for sending us our savior, Nate Ravid. Nate showed up a half hour ahead of schedule this past Tuesday, and stayed with us until 4AM on Wednesday.  He was extremely knowledgeable, calm, and got the job done!  We successfully restored from backups, rebuilt our Active Directory and two domain controllers, a print server, and manually rejoined more than 50 computers to the domain. ALL before 8am when school started the following day. On Wednesday (February 15th), every class went on as scheduled with the technology they needed. It was my mission to make that happen and WE DID IT! Pretty incredible if you ask me! Thanks again Nate. It was a pleasure having Modern Pizza  and cookies with you at midnight. I apologize if you didn’t spend Valentines day with a special someone because of it.

8) Bob Cashman of CCA and his consultant genius Dana Urban, who answered my dire call for help at 6pm on Wednesday (after our all nighter). Our email server was the last of the casualties and Victoria and I were brain dead at this point. These two men spent 4 hours on the phone helping us trouble shoot our issues and get a plan together for the morning. Their help, along with First Class Support, guided us to the best case scenario possible on Thursday morning. We had our email server up and running by 11am.

9) TechnologyPartners, our online backup company who answered countless phone calls helping us restore all servers with minimal data loss.

10) Andrew Speyer, Director of IT at Choate, one of my CAIS CoT best buds,  who answered my “HELP!” text within minutes when I needed a contact for a fiber technician.

11) The many patient students and faculty of HH, whose endless cheers and words of encouragement and appreciation kept us going. Thanks for not “giving up” on technology and for realizing that sometimes these things can happen. (Heck if some of the big companies (Google, Twitter, Facebook), with unlimited funds can have network issues, we’ve done pretty well over the years!)

12) And last (but not least), my Head of School, Bob Izzo,  who was also encouraging, patient, trusting and actually joking with me through all of this. I appreciate his support and I feel lucky to have him as my boss. He could have been  A LOT less patient. And I thank him, in advance, for listening and supporting our proposal for upgrades to our infrastructure 😉

So, if you stuck with me through all of these thank yous, I ask that you follow these amazing people on Twitter or LinkedIn, support their businesses, read their blogs, etc. They are an amazing bunch and I am eternally grateful.  I hope I didn’t leave anyone out. If I did, please forgive me. I am still catching up on my sleep! It was a wild few weeks (especially this one). I truly hope that bad things happen in 3s. It would be nice if the rest of the year was smooth sailing!

Advertisements

7 Comments

  1. I second all those thank you’s to the many people who went above and beyond to help us out. This particular week was a 2-3 man job and being that I am the only server/network technician it was great to have such knowledgeable and supportive help from everyone. I would personally like to thank Lorri Carroll for her patience and support and for helping out and staying with Nate on Tuesday night while I went home for a few hours to check on my baby. When I came back in on Wednesday morning, I was more than surprised that she rejoined the 50+ computers to the network. She is an amazing boss, an inspiration, a mentor and a great friend. Taking this job at Hamden Hall about 4 years ago has changed my life and I couldn’t be happier to work with such wonderful people.

    Reply

    1. YOU, my friend and colleague, are amazing; you are one of the strongest women I know! And… you are a good teacher (heck I needed to re-learn how to join those computers)! You juggled so much (home, baby, etc.) and got us to where we needed to be. You could possibly rival me (and a few others) for the top prize of workaholic. Even after everything last week, you worked YESTERDAY from home, on your day off. You taking this job at HH 4 years changed my life too!

      Reply

  2. Wow! This whole post, all the words and all the stories don’t surprise me. That is an incredible team there at that incredible place, Hamden Hall. The spirit, tenacity, and leadership with which you tackle issues is unmatched. You know how to get the job done and you make all those around you want to do the same. Thanks for sharing your woes, but what I really read is one more illustration of why I miss your leadership, guidance and incredible work ethic in my everyday life. I knew it when I was there, andcontinue to appreciate the place I called ‘work’ for so long. The close colleagues I had at Hamden Hall changed my life. Thanks in so many ways…

    Reply

    1. Ginnie- Thanks so much for sharing that! It has been a wild few weeks! We do have some incredible, dedicated people at Hamden Hall and, I agree, it is a special place. I miss you very much and hope you can stop in for a visit soon. Please let me know when you will be up north again!

      Reply

  3. This is the first blog I have read. I was very impressed with the information & how exciting it was to read. I know, from experience, how much of a family Hamden Hall really is. Thanks to all who helped me understand the ins and outs of the internet, as well as the chewings that went on during the 2 weeks. I think I’m even glad to be part of this internet of ours.

    Uh Oh–I don’t know how to send this!

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s