VMware: Famous Last Words

This is one of those Incredible Coincidences or possibly Acts of God.  I can’t tell you which or whether.

Last night a coworker and I gave a presentation to our local linux user group on virtualizing the enterprise with VMware.  It went reasonably well, in that nobody threw anything and the only people who fell asleep did so on the presenters before us.

One of the key points of our presentation was the solid reliability of VMware, especially of Windows servers under VMware: they just run.

Mostly.

——————————

Yesterday I had to make an emergency trip with a different coworker to a business unit ninety minutes away.  It was way further than we normally cover but it was day two of the emergency and needed another set of eyes (or four).  This project was an unmitigated disaster and had been so for the greater part of a week, after a server failure.

My job was to make it work, by hook or by crook, before I left for the day.  In spite of my confidence, the project desperately continued its course of failure, almost up to the point where I had to leave so I could make the VMware presentation.  Ironically this project involved the new VMware ESX4i server.

Fortunately everything came together and I made the presentation on time, had dinner at 10pm, and was in bed by one am.  I had not seen work or personal email in twenty-four hours.

——————————–

I was rudely awakened this morning by my wife’s cell phone, which she uses as an alarm too.  It went off, making an unholy racket, ten minutes before my alarm was supposed to go off.  That’s ten minutes of snoring I will never regain. Since my wife has given up sleeping almost permanently, there was no one but me to turn it off.

Downstairs I went, building up a good head of steam to describe what I was going to do to her cell phone, when she pre-empted me because MY phone had been going off like mad.

When I looked, there were two hundred and twenty two text messages from my network monitoring system, starting at three in the morning.

This was not good.

———————————

On the ride to work, there were three calls about the network being down, off, odd, or otherwise unworkable.  This is not supposed to happen until we have our morning coffee or Mountain Dew (or crack).  And I was the beneficiary of another one hundred text messages from the monitoring system.

Every part of the network was coming up and going down like a rollercoaster.  We were dumbfounded, as there was no apparent cause, so we dug in.

What event precedes an awful lot of trouble?  A forced Microsoft update.  Sure enough, several machines had rebooted.  It was Bill’s Will.  These machines were set to NOT reboot but it was Bill’s Will.

We located all sorts of problems and my boss became all sorts of helpful.  When I say helpful, I mean my boss hovered, paced, and stared over everyone’s shoulders like a madwoman.  She started making helpful suggestions, like why not call the storage vendor.

The correct answer to that question is `because it’s not a storage issue’.  Never one to be deterred from being helpful, my boss called the storage vendor and set up a service ticket via phone.  We lost a solid hour on the phone while the storage vendor told us there was nothing wrong with the storage.  Now that was helpful.

Apparently the chorus of ten voices chanting `VMware, VMware’ set up my boss for another Helpful Idea<tm>: why don’t we call VMware?

Why didn’t we think of that?

To only slight surprise, I got on the phone pretty quickly with VMware India.  It took them forever to put in the ticket because their system was down.   VMware put in some quality time and quality service.  They went throughout the system, trying to figure out what happened and how to fix it.  It was most impressive.

What was less impressive during the several hours spent in this endeavor was my boss, being helpful (surprise!).   Standing right over every single person who got on the phone with Support.  Pacing in the five square feet of space left in the network closet… err…  room.  One coworker almost had to be physically restrained from whapping the boss verbally (hopefully not physically too).

By two pm, still with no email or breakfast in sight, I handed the phone to someone else so I could get some breakfast.  My boss took this as a cue to follow me back to my desk, question me a bit, then keep bouncing between my desk and the network closet…. errr… room.   By this point, most of my coworkers had developed what I can only describe as some sort of Spontaneous Tourette’s Syndrome, complete with muttering and spurious cursing.

One bite into my breakfast, She Who Must Be Obeyed calls me over to another workstation, where under her direction, a colleague is restoring a copy of our domain controller to another machine.  Somehow managing not to allow my eyeballs to start rolling like a slot machine, I put on my Serious Look<tm>, while my colleague explained to her that it would take 2-3 hours to complete the restore.

It was at this point that I came up with one of my Positively Brilliant Ideas<tm>: for the next emergency, we would have to put together the SDS - the Special Distraction Squad.  This would be a special team, selected for their technical ability and technical bullshitting skills.  Their sole job would be to keep the boss occupied, thereby allowing the professionals to do their jobs, all without benefit of help and great ideas from the boss.   One idea we were kicking around was a series of screens that look like installation screens.  We’d tell her we needed her to sit by this pc and monitor the process of the installation, then report back to us when it was complete (hours later or until we got the problem fixed).

Meanwhile I ferried news and refreshments between locations, doing my best impersonation of Lance Armstrong in slow motion, running through the halls (in slow motion - you had to be there).  After all, it’s not a matter of what the problem is, it’s a matter of what the perception of the problem is.  If you run through a place of business, especially one that’s so laid back you have to check for a pulse, people perceive that you’re doing everything possible to fix the problem (I’m learning about the public relations aspect of MIS these days).

So while my colleague was Boss Babysitting, others were on the phone with Support.  Shortly thereafter, I noticed virtual servers coming up.  This was tremendous news! In fact, everything came up.  Well, everything except email.

So we had only lost about seven hours of productivity.

At one point there was no internet.   Have you ever seen two hundred and fifty people in an office with no internet?  It’s not pretty.   In fact, it’s not pretty with internet.  You’d have thought they all had their puppies kidnapped.  Some were still bent out of shape because their Blackberries weren’t working.  Of course their Blackberries weren’t working - the mail server was down.  It got a lot more quiet once we brought the internet back up: they could get back to their Important Business Faceyspacing with all due haste.

We tested and announced that the servers were back up to an almost empty building.  Everyone had gone home except for us and Payroll, the poor bastards.

Email would not cooperate, no matter what we did.   So when all else fails, Call Microsoft.  Microsoft has this great plan by which you can call and get support for only two hundred fifty dollars, somewhere in India.  Today they were running a special: it would only take them about two hours to call us back to work on the problem.

By this time there were only two of us left.  Even the pizza delivery place laughed at us.  The ironic bit here was that the pizza delivery window was only one hour (and considerably less than two hundred fifty dollars, thankfully).

You can’t accuse the boss of being stupid: she left hours ago, understanding that perhaps she wasn’t going to be helpful too much longer.

I called my wife and told her I wasn’t going to be too helpful in terms of dinner.  She sounded appropriately sympathetic (and I didn’t hear any cheering in the background so I assume she was sincere).

It’s 8:30 and I’m banging on the keyboard while my brave coworker talks to Iqbal from Microsoft India.  There’s nothing I can do but avoid `helping’ him the way my boss helped all of us today.  There aren’t even any desks to drape silly things over or turn upside down for fun.

Forty eight hours have elapsed since I last saw my email (twenty four since I had a non-emergency related conversation with someone).

I miss my dog.  And my wife.  Even the cat.

Tomorrow is when the Real Fun starts.  Cue finger-pointing and Executive Blame Sessions.  The meetings will continue until morale improves.   And to think, I only do this because I love computers (and to pay for health insurance, but don’t get me started on THAT).

LESSONS LEARNED

• Never, NEVER give a presentation and claim something works
• As soon as there is a network issue, deploy the Special Distraction Squad
• Invest in freeze-dried pizza
• Work in a department that always goes home at 5pm, regardless
• Do not have sex on your desk while others are around (unless you like that sort of thing)