When the network is down…

Here’s my list of 10 things I learned when the network was down.

http://www.techsling.com/2011/03/5-reasons-to-develop-a-disaster-recovery-plan/

1)…having a good, well tested (and often), disaster recovery plan is BEYOND necessary!

2) …even if you don’t have all the answers yourself, if you surround yourself with the people who do, the job gets done!

3)… there is a weird sense of “quiet” on campus.

4) …some educators can be flexible, creative, go with the flow, adapt lessons and pick up another tool. Yes, it might be less than ideal, but the students will still learn each and every day. These educators, without even realizing it, seized the opportunity to model essential 21st century skills  to their students.

5) … some educators seize the opportunity to say:  “You see! This is why we can’t go to e-texts/ use blogs/ wikis, etc. This stuff NEVER works”. Imagine if we said this about cars when they broke down. We’d be walking a lot!

6) …some people understand that sometimes these things happen. Others don’t.

7) …it’s very easy to be a “Monday morning quarterback”. “Shoulda, coulda, woulda” doesn’t really help.

8) …some students are able to adapt. They used the resources they had (ex. their home email, texting, Facebook, cloud based applications, etc.) to complete what was asked of them. YAY!

9)…some students are NOT able to adapt. They used the network being down as an excuse to do NO work, which gave the teachers mentioned in #5 another reason not to use technology.

10) … you learn that money can’t solve all problems, but it helps with a lot of them!

“Bad things happen in 3s” …???

Whoever said this, I hope they were right. Here’s a peek at the network woes that have plagued Hamden Hall in the past month.

The mice at Hamden Hall must have needed some more fiber in their diet!

1)  Bad network hub caused a broadcast storm that took down our network. After a few different methods of troubleshooting, we were able to locate it the “old fashioned” way… we unplugged the troublesome ethernet cord from the port on the switch and waited until someone yelled. We  replaced that darn little NetGear hiding in the Art Cottage.  (Yes, I know we should know better … NetGears are famous for that!)

2) The following week, a mouse ate through the fiber that connects our servers to the rest of campus… Those darn little teeth! It was repaired and resolved. We were down for about a day and a half. (Email worked from home though, thank goodness.)

3) The last, and most challenging of all the issues: Active Directory DNS database corruption. This problem became noticeable between events 1 and 2. We attempted to repair and “band-aid” the database as much as possible until the band-aid couldn’t hold it any longer–this happened Tuesday (Happy Valentine’s Day) at about 2pm. We worked through the night (literally) into Wednesday morning to rebuild 3 core servers to ensure that the problem had been resolved without any band-aids!

Now– we have had some unique issues at Hamden Hall before. Worth noting is the  pipe that burst in the server room on the first day of school three years ago– water shooting directly into our core switch and trickling into the remainder of  the undeployed PCs. We put those wet PCs out behind the server room on a pallet to dry in the sun, only to be hit by a car coming around the turn of the drive way.  Who can make this stuff up? But never have we had 3 unrelated issues hit us this close together to cause such an interruption in teaching, learning and productivity. (I will reflect a little more about this in a different post. )

The purpose of this post is to publicly thank all of those people who helped to get us to where we are today.

1) My Network Manager, Victoria Helland, who gave 110%  trying to support and resolve all of these issues. She truly carried the burden of the network outages on her shoulders and she did everything humanly possible (and without much sleep) to get us up and running. I am lucky to have her on my team!

2) My Husband Jon, who not only was an amazing support to me emotionally through all of this, but was full of what seemed to be unlimited resources when it came to knowing the right people at the right time. I am lucky to be married to him and grateful for his vast experience in the IT field throughout his career.  Not only did he hook us up with some great people, he called them to make sure they took good care of us. I also thank him for handling all the drop offs, pick ups, homework, cooking, and for packing me everything I needed to start the day on Wednesday (after the all nighter). The outfit matched (I was a little worried). Thanks for checking on me all through the night, listening to me, asking the right questions to help me troubleshoot, and being a shoulder for me to cry on when I finally lost it. Sorry for missing Valentines Day. I love you.

3) My kids, who listened to my endless conference calls on speaker phone on the way to and from school and who were able to “go with the flow” staying late at school, and coming in early. They are strong and independent (YAY!) and I am very proud of them!

4) Jim Hunter, Director of Facilities, who always goes above and beyond to support the tech department. From coming in at 5am to help the fiber technician pull the new fiber in the dirt basement to delivering endless food, coffee and words of encouragement. He truly epitomizes the Hamden Hall spirit of commitment and dedication. We couldn’t have gotten where we are today without his help. His consistent “bright side” thinking helped us all plow through the issues.

5) Chris Buckley and Sarah Ludwig, who were on the front line supporting teachers and students through all of the ups and downs. You did this with a smile and patience. I am lucky to have such a great team!

6) FiberOptics Plus (a referral from my husband Jon), and especially the tech Matt,  for quickly getting to campus to figure out where the break in the fiber was, and for coming back to campus at 5am the morning after the SuperBowl (Go Giants!) to get us working by 9:30am (3 hours ahead of the original projection).

7) Brian Diamond, President and CEO of LANStatus: “Exceptional Attention. Extraordinary Results”– very true! (also a connection from Jon)  Thanks, Brian, for sending us our savior, Nate Ravid. Nate showed up a half hour ahead of schedule this past Tuesday, and stayed with us until 4AM on Wednesday.  He was extremely knowledgeable, calm, and got the job done!  We successfully restored from backups, rebuilt our Active Directory and two domain controllers, a print server, and manually rejoined more than 50 computers to the domain. ALL before 8am when school started the following day. On Wednesday (February 15th), every class went on as scheduled with the technology they needed. It was my mission to make that happen and WE DID IT! Pretty incredible if you ask me! Thanks again Nate. It was a pleasure having Modern Pizza  and cookies with you at midnight. I apologize if you didn’t spend Valentines day with a special someone because of it.

8) Bob Cashman of CCA and his consultant genius Dana Urban, who answered my dire call for help at 6pm on Wednesday (after our all nighter). Our email server was the last of the casualties and Victoria and I were brain dead at this point. These two men spent 4 hours on the phone helping us trouble shoot our issues and get a plan together for the morning. Their help, along with First Class Support, guided us to the best case scenario possible on Thursday morning. We had our email server up and running by 11am.

9) TechnologyPartners, our online backup company who answered countless phone calls helping us restore all servers with minimal data loss.

10) Andrew Speyer, Director of IT at Choate, one of my CAIS CoT best buds,  who answered my “HELP!” text within minutes when I needed a contact for a fiber technician.

11) The many patient students and faculty of HH, whose endless cheers and words of encouragement and appreciation kept us going. Thanks for not “giving up” on technology and for realizing that sometimes these things can happen. (Heck if some of the big companies (Google, Twitter, Facebook), with unlimited funds can have network issues, we’ve done pretty well over the years!)

12) And last (but not least), my Head of School, Bob Izzo,  who was also encouraging, patient, trusting and actually joking with me through all of this. I appreciate his support and I feel lucky to have him as my boss. He could have been  A LOT less patient. And I thank him, in advance, for listening and supporting our proposal for upgrades to our infrastructure 😉

So, if you stuck with me through all of these thank yous, I ask that you follow these amazing people on Twitter or LinkedIn, support their businesses, read their blogs, etc. They are an amazing bunch and I am eternally grateful.  I hope I didn’t leave anyone out. If I did, please forgive me. I am still catching up on my sleep! It was a wild few weeks (especially this one). I truly hope that bad things happen in 3s. It would be nice if the rest of the year was smooth sailing!

We owe it to our kids!

Yesterday, I gave this  presentation to the Middle School.

Think Before You Post!

I started with my favorite AdCouncil PSA “Think Before You Post”. These 30 seconds really capture, so powerfully,  what can happen with an image online. I then moved to the topic of Facebook privacy settings, and I think I opened a few student’s eyes about what the setting “Friends of Friends” really means. And, as expected, our students are really becoming active on  Twitter (we are not using it at HH for academic purposes as of yet, but students are definitely feeling out the “social scene” there). When I asked students about this, they said “We went to Twitter because our parents are on Facebook”.  Hmmm… Interesting!  This led to a great discussion about  how public Twitter is… and I showed them how easy it is for me (or anyone) to see what they are tweeting, especially when they mention specific key words or hashtags (and especially the words “Hamden Hall”). “I don’t have to dig”, I said.  “Hamden Hall is a column on my TweetDeck. It’s our “brand”, we need to protect it.” I think they got it.  We talked about how private “protected tweets” are and I also mentioned the situation with a student at Don Bosco Prep (who “protected” his tweets and yet, with 1500 followers, how “protected” are they?).  This segued into a good discussion about digital footprints. I showed them 4 ways to check their footprint (from this post (The Innovative Educator)).  I used the example of “Hamden Hall”, instead of a real person, to demonstrate the results of these sites. In preparation, I tested this right before the presentation to be sure that no one in the room would be embarrassed by the content. Not 20 minutes later, when I used Spezify, a brand new tweet showed up that was not very positive about Hamden Hall. On the one hand, it was a little uncomfortable that a very inappropriate word was on the screen in front of the students, and I had no idea who tweeted it, and yet… it was a PERFECT example  what I was trying to show could happen. I calmly said, “you know what I’m doing when this assembly is over”. I finished up… and the students were buzzing! I jumped onto my twitter and confirmed what I had hoped to be true– the tweet did NOT come from a Hamden Hall student. Phew!

By the middle of the afternoon, the Middle School Dean and Middle School counselor were sending positive feedback my way. The kids were talking about how they were going home to Google themselves and check their Facebook privacy settings. YAY! They were also talking about the inappropriate tweet– (and wondering if we “caught” the person)  All and all, a very positive experience.

Next up was a 7pm presentation to the Middle School Parents. While not as well-attended as we had hoped, all the parents who came said that their kids were talking about the assembly when they got home from school. I went through the same presentation with the parents, answered questions, shared resources, clarified some information and we ended with coffee/cookies.

Overall, I think it all went extremely well. Honestly, though, I am disappointed that more parents didn’t attend the discussion. Here’s my take on it:

We spend a lot of time teaching our kids how to cross the street safely. We teach them how to look both ways, we practice it over and over again (FOR YEARS). We  hold their hands for a long time until we are comfortable knowing that they can do it on their own. We need to do the same with kids online… we can’t send them out there without teaching them, guiding them, supporting them, and answering questions along the way. We owe it to them!

Here are some of the resources that I used for my presentation:

How to Check Your Digital Footprint

Controlling Your Digital Identity

Building an Online Presence is More Important Than Ever

ChildrenOnline.org

“When Saying No To Facebook Isn’t An Option”

Don Bosco football star gets expelled…