RTFM: How to keep CIOs under control

With apologies to The Register’s Bastard Operator From Hell series (originals here). This post is a satirical response to the recent iTNews article ‘Death of the SysAdmin’.

It’s Tuesday, which is one of my favourite days. Every Tuesday morning I set the index files on the Samba server to rebuild. It pipes the output to a nice little status bar I’ve built in EMACS, ticking happily along with happy asterisks. While I watch it out of the corner of my eye, I always start the morning by making myself a fresh pot of coffee and checking the server logs.

None of the users can log in, of course, while the index files are rebuilding, but that’s the price of serenity, and they got used to it eventually after I calmly explained how their files would be deleted each week without the re-build. The new version has fixed the login bug, of course, but if I applied the patch, what would I do on Tuesday mornings? You see, working as a systems administrator is a vicious cycle of stability.

But then the calm is disturbed by a phone call. I take a sip of my double filtered Lavazza and pick up the handset. Jesus. Someone’s already calling me and I haven’t even had to the chance to check the CEO’s browser cache for that lesbian porn he’s really into. What gives? “You have to do something!” the urgent whisper comes. It’s Barry, the database admin in accounts. “What’s wrong?” I ask calmly. Barry’s a bit flightly at times. Comes from dealing with Oracle too much. It’s the hard drugs they give you at OpenWorld — they mess with his mojo. “It’s Vito,” he says. “He’s making all kinds of wild claims.”

I think back. “Vito from Worley Parsons? I thought we handled him years ago.”

“Yes,” says Barry. There’s a pause. I imagine him putting the phone down and poking his head up above his cubicle, checking for anyone listening in. Mentally I will Barry to calm the fuck down, channelling Larry Ellison meditating in his Zen garden, twin 22 year old bikini babes sitting with him on the Zazen mats on each side, their bleached blonde hair rustling gently in the breeze, appendages gently bouncing. Tranquil.

“We did handle him,” says Barry. “But now he’s moved on to Fortescue and he’s talking to journalists again. He’s …” (at this point there is another pause. I hear a few mouse clicks and then Barry starts up again) “… talking about driving a transformation program to align and then proactively deliver IT/IM services to sustain and grow the business. He wants to build effective teams that can meet growing demands by being innovative and not accepting the status quo.”

“Guh!!” I spit a mouthful of coffee out, straight onto the keyboard. Damn. That’ll take a while to clean up. I make a mental note to get the Junior Windows Intern to do it. Surely he can manage that? But then …. maybe not. I mean, he DID study Windows Server — an oxymoron if I ever heard one.

“It gets worse,” says Barry. “What could possibly be worse?” I ask. “This guy just keeps on getting away from the herd. Doesn’t he know that systems administrators are in charge, and CIOs are just there to keep the bean counters and pointy headed bosses off our back?!”

“He’s talking about the cloud,” whispers Barry, his voice frantic now. “He says, in an article this morning, the days of the traditional sysadmin are gone. He says we’ll be replaced by cloud computing, with all services to be provided through the cloud.”

A joint silence enters the conversation as Barry and I contemplate this monstrosity. Not only has Vito gotten away from the herd; he’s gone rogue. This will need to be dealt with quickly. I realise I must immediately bring in the big guns: The AusNOG list. “I’ll handle it,” I tell Barry. “Calm down and reboot the Exchange Server a few dozen times. You know that always makes you feel better.”

“I knew you would know what to do,” says Barry, whimpering a little. He hangs up. I make a mental note to scale down the VMware instance on the production transaction server. That’ll make him think the database is underperforming, a perfect excuse to go into a 24 hour orgy of table optimisation and self-flagellation. Database admins are all the same.

Immediately I put out a mass broadcast on the AusNOG list. “Error code 25 at Fortescue,” I write. “Inappropriate ioctl for device. Anyone know how to handle this one?” A reply comes back immediately. “Affirmative, roger,” replies a long-time contact, Skeeve Stevens. “I’ll handle it. Your line?” “Yes, please, I respond.”

Across town one of the brothers at Fortescue will be receiving a system message on his desktop FreeBSD box from Skeeve. We all maintain limited remote login privileges on each other’s desktop systems (without root access, of course — you don’t want to be crazy) for exactly these kind of emergencies. “The log files on the HTTP server are filling up with bullshit,” it will read. “Concatenate the file to /dev/null immediately, excess output to overflow”. Overflow is my code name — I let one of the coders install a virtualised Windows Server 2003 instance on one of the development boxes once. The crew has never let me forget it.

Across Fortescue’s operations servers will be powering down, or simply entering infinite reboot loops, shell scripts set up precisely for this purpose coming to life like a million tiny worker ants, their eyes gleaming feverishly in the dark datacentre night as they go about their business. First it’s the email server (an easy target, they run Exchange), then the network file server, directory, app servers for HR, CRM and ERP, and even the IP-PABX machines running the IP telephony network. The print servers stay up, but they auto-randomise the output so suddenly, everyone’s print job is being printed a floor above or below from their location. It’s all perfectly synchronised, like a thousant ballerinas stepping in tune. A well-set up system.

I tip back my chair and start up a Nethack instance in a terminal. I’ve just skilfully avoided several obvious traps and am about to retrieve the Amulet of Yendor when the phone rings again. Annoyed, I turn away from the terminal and pick up.

“Tech support,” I say.

“This is Vito,” a deep voice on the other line says. “Vito who?” I reply. “The chief information officer!” the voice on the other end replies. He’s obviously taking this too seriously. “Is this the Sydney datacentre operator?”

“Vito Corleone?” I reply. “I saw that film a while back. Not bad, although I thought the accents were a little overdone.” “Stop playing around,” Vito says. “We’ve got a serious outage on our hands. What are you doing about it? The CEO is breathing down my neck as we speak!”

I take another sip of coffee and ponder my next Nethack step. Vito has interrupted me at a critical point. “Don’t worry,” I say. “None of the servers can actually go down. We’ve virtualised the entire datacentre. It’s in the cloud.”

“What?!” sputters Vito. “But we haven’t even completed the strategic migration plan yet. Who approved this?!”

“It has been on the cards for a while,” I respond. “By distributing the total computing load across several different nodes in disaggregated virtual machine environments, we were able to increase uptime, boost the efficiency dividend and implement an open-architected mission-critical collaborative environment, based on dynamic run-time principles which iterate automatically.”

There’s a pause on the other end of the line. “Go on,” says Vito cautiously.

“It was the obvious thing to do,” I add. “We had to tackle the problem of enterprise-wide grid-enabled complexity with respect to the service oriented architecture. And of course there was the issue of the triple-buffered background throughput ratio. The solution-oriented extranet wouldn’t have been able to handle the virtual actuating synergy array, otherwise — not to mention the on-premise private cloud paradigm.”

It seems like Vito’s catching on. “But … what does that actually mean?” he sputters. “The servers are down. Can’t you fix the problem?”

“What are you talking about?” I ask, still calm. All those years of dealing with CIOs imbibes a certain mental fortitude — like the kind you get after dealing with a screaming baby. They never seem to understand. It doesn’t matter if the servers are up or down — that’s a user problem. The real issue is whether they are configured properly in the first place. The system must be perfect, pristine. Users pollute that nirvana. “We don’t have servers any more — isn’t that obvious?” I say. “We have a fully-configurable, modular open architecture, built on top of a triple redundant contextually-based data warehouse. Isn’t that what you wanted?”

On the other end of the line, I hear a muffled sob, then a pause. “I’ll call our outsourcer,” Vito says. Looks like his fighting spirit is starting to pick up again. “CIOs these days are brokers of services. I can get this fixed!” “No worries,” I say. “Make sure they know we work to a demand-driven dymanic methodology.” *click*

I go back to my Nethack game. Ten minutes go by and I’m just about to switch to my afternoon task — setting up an auto-leave-deny script for the HR platform for any employees who have filed a helpdesk ticket in the past six months — when the phone rings again. “It’s Vito,” the shaky voice on the other end of the line says. Good, he’s able to find a phone number for himself — he’s obviously asked his PA. A smart one. “Is this my account manager?”

“Sure!” I say, in as cheerful a voice as I can manage, given that Vito has interrupted my important sysadmin work once again. CIOs are like children, really. Pity how any call he makes will redirect to my desk phone. Ah, routing tables. So simple, yet so powerful. “How can we help?”

“You again?!!!” sputters Vito. “I thought you were the datacentre operator?” “I am,” I reply. “Your ‘right-sourcing’ strategy saw several roles integrated and re-contextualised to better highlight the potential of a reactive collaboration style. I’m based in Malaysia.”

“The CEO has emailed me five times in the past half an hour,” whimpers Vito on the other end. His spirit is clearly broken. “Isn’t there anything we can do to get the servers back up?”

“Well … the old servers are still around,” I muse. “We haven’t decommissioned them yet. They’re still in the datacentre — do you want me to call the systems administrator?” “YES! YES!” exclaims Vito. “Call him! Turn them back on! The cloud doesn’t work!”

“We might need a few extra hands,” I say cautiously. “Say, five extra sysadmins on full salary, two Oracle DBA admins, some SAP consultants. But it’s OK. Now that we’re not using the cloud, we can retrench the entire Microsoft admin team. We’ll base everything on Unix. It’s much more stable that way. And you’d better issue a statement informing the press that we’re dumping the cloud model. Tell them that there are concerns around the legality of storing data offshore. They’re dumb enough to believe that.”

“Yes,” whimpers Vito. Putty in my hands. “Do it. I’ll inform the press. Report to me when you’re done.”

I fire off a quick message to Skeeve. “Physical address space upgraded,” it reads. “Increase file allocation. Delete VFAT table.” “Confirm,” he replies.

And, with that, another arduous day as a sysadmin is over. We never receive any credit for our work. We just toil away, day in, day out, keeping the peace and fighting for the cause of justice and truth. One day we’ll be rewarded for our labor, and everyone will realise that without us, the world would shudder to a grinding halt. But that noble day has not yet come. So, until then … it’s time to rebuild the index tables again and make another pass at the Amulet of Yendor. Life is good.

Image credit: azbarkmans, Creative Commons