work/musings { Of Enterprise and Workgroup

Long have I pondered the differences of the quality of those who take up the command line and delve into the depths of Systems Administration. They are a varied lot, while not exclusively male, this is the majority, and one I'm happy to see it changing, if albeit slowly. The spectrum of personality types, ability levels and discipline are included under the label.

Many are called, few should stay.

First a clarification of the term 'Systems Administration'. It is 'Systems' with an 's' at the end because the profession is, not as popular opinion seems to hold, about management of complex systems of interconnected entities. Not merely management of one technology type. I have found in my journeys that too many junior, and often many senior, Admins confuse this fact. Technologies come and go, tech companies rise and fall, but through this all, the basics of the profession remain the same.

1. Its about people as much as it is about technology. For without customers we would not have jobs.
2. Simple is better. More with less.
3. Documentation will save you.
4. You are always learning.

Really that's pretty much it. While many SAs like to view themselves as Wizards (myself included) weaving mystic incantations of digital power (which is cool imagery and I am a D&D nerd), in reality we are simply mechanics and custodians. Architects and builders. Systems Administration is not something that can truly be taught. You can go to school to get theories and basics of computer science, but like most professions, you learn the most on the job. I've met a few in my career who are incredibly smart, good with computer science, but don't have the chops to be a SA. Some of the best SAs I know never went to college. Its more than being able to code a program. Its more than being able to troubleshoot a problem. It requires attention to detail. The ability to take requirements and meet deadlines. It requires sacrifice, for you will be salaried and will be called to work long hours. There are rewards and accolade, but mostly your work will go unnoticed, which is not necessarily a bad thing, as if you do your job right things go smoothly and is transparent. However this transparency comes with a down side, the ones who control the money assume that things running smoothly means we as an institution can do more with less. That technology budgets can be cut and skimped. So in this respect a SA must be part cheerleader, promoting the environment. Providing uptime metrics, upgrade and technology refresh schedules. Budgeting quotes. You must be able to talk with others outside your group in terms that people not familiar with the field can understand. Accountants, upper management, and sometimes lawyers.

In my travels I've also realized that we can be broken down into three categories. Well four, as there is also the 'Contractor', but I will not get into the life of a digital mercenary, and really what they do is fill in one of the categories below for short-term or per project. Now within each category there are levels of experience from Junior Admin to Senior+ (I use these because the job title and ranks are different from institution to institution, there is no commonality. In some places the SA is a Systems Engineer, in others a Technical Analyst, in a few older shops they still carry the title Systems Programmer, in a few newer places Systems Administrator, I even had one that was Systems Administrator/Engineer/Architect Lvl 4).

The first category is Desktop Support Admins. These are the people interface with customers the most directly. They go desk to desk fixing workstations and installing new ones. They can manage as few as one desktop or in some of the largest places 30,000+. Their world seems strange and hectic to me as my whole career I've been primarily on the server support side, which I will break up into the last two categories.

The next category is Workgroup Support Admins. Workgroups are generally smaller units or departments within an institution, or in smaller companies could be the whole IT department. I categorize this group as a place with less than 10 servers. These Admins often spend their time spread around among a variety of tasks, providing support for almost everything from the desktops to the places backup and recover system. They are often alone or on a very small team. Often they are on call all the time and can have either a lot of time to work on the future of the environment, or they can be hopelessly behind and struggling to keep the environment running, let alone stable. Documentation is vital. I've never been a Workgroup Admin, but many of my friends and colleagues in the IT world are or have been.

The last category is the Enterprise Support Admins. This is what I've been for the last 11 years. The Enterprise I classify is really anytime you get more than 25 servers and usually have a dedicated data center, raised floor, cooling, special power setup and functionality of the mechanics of running the environment (such as the network, backup/recovery, storage, etc.) are separated out into their own team. Sometimes Admins service multiple teams for example my last job the UNIX team was also the Storage Team and the Backup Team. The economies of scale start to really be felt here. Generally effort to support systems is a logarithmic curve, not a geometric one with a plateau. Its multi-variable. The steepness of the curve depends not only on number of servers but also number of admins, quality of process, ability of support/peer services, and so on. If you have well documented processes and procedures, an orderly way to get problems into and out of the team, and an enforced change management process, four SAs can manage and support 30 times their number in systems. My old job after much toil, we created an environment where four admins on the Solaris sub-team (we broke UNIX support down by OS specialties to handle the workload) supported 150 OS instances. I've not yet worked in an environment of less than 300 OS instances (by instance I mean both an OS running on a single piece of physical hardware and one running as a virtual instance in a logical OS space). Generally you're dealing with millions of dollars in institutional assets.

At all levels you have to deal with politics. Intra-team and inter-team. Between managers and colleagues. Some managers jealously carve out their own empire and defend the borders (and budget) of it against all comers. Some place their people up on pedestals and treat them like divas. Some exert authoritarian control and micro-manage them with an iron fist. Others are ineffectual or try to be everyone's friend. Some styles work, some don't. SAs have a reputation of being as easy to manage as a 'herd of cats.' But this could almost be said of any profession. Hell the waiters at the food service job I had one summer in high school were the same way.

As everyone is different, some people do well in the Enterprise where you must work efficiently with others and have procedures that move like clockwork or else drown in a sea of work. Others prefer the often wild west one sheriff in town life of Workgroup support. And yet others like the customer focus of Desktop Support.

However my experience has been there are few good Systems Administrators. I will go on explain some of the common problems I see. I know I fall into one of them. Against nothing surprising, these types can exist in any profession.

1. Shiny Object Syndrome - this is what I call a phenomena I see where an Admin is distracted and unfocused because he/she wants to work on the latest technology, or the latest project, or the latest high profile problem, often to the detriment of their assigned project load. They are a load on their team because that project work has to be done and often management, when its reached a critical point, shifts it to another admin.

2. Negative Nelly - nothing will ever be good. No technology does everything you want it to or was designed by a 'pack of retarded apes' (I've actually heard that phrase used). These admins throw up a sphere of negativity that makes others seek out an alternative place to get their support. They are a load on their team because their sour mood is often infectious and it can sink morale.

3. Someone Else's Problem/The Teflon Admin - these Admins find ways to shift any problem and any project on to others. In some cases they shift the work, but then take credit (like some bad managers do) for the work of others.

4. The Burnout - these Admins fried themselves. Either working too much, caring too much, or letting it too personal. They have now become a load on their team because they are depressed or frozen into inaction.

5. The Know It All - really, no one likes a know it all.

6. Lost in the Past aka The Historian - these admins are stuck in the past. Everything is compared to some, usually archaic, technology that is long gone. Or they never change because 'this is how we've always done it' or 'I've done this once and it didn't work/it hurt me so we should never try again.' Paying homage to the past as a point of reference or a place to learn from mistakes is fine, but dwelling there is detrimental.

7. The Family Guy - no not the hilarious cartoon. This is the Admin that is 'family focused'. They work a set amount of hours in order to 'go pick the kids up' or 'spend time with the family'. How is this bad on the team? At my last job we had a 'Family Guy'. When he was on call, he would often redirect a support call to another Admin, who was not on call, because he was at some kind of church function, at the movies with the kids, etc. In short their work doesn't get done or gets shifted to other Admins. I've also known many Admins who have managed to balance work and family and not shift their work load to others. I used to think making a commitment to be family focused was a good thing, I didn't (and still don't) have kids and thought it was important, and first didn't mind picking up the extra work so someone could see their kids. This ended when I became a Lead, and saw the stress it put on the rest of the team. Many of whom also had families but would have to help pick up the slack.

I could go on, but as I've stated, these are personality types common in most professions. Or at least I believe so, but I'll admit, my job experience has been limited to mostly IT centric.

One last topic, the differences between Junior and Senior Admins, as I've noticed many do not seem to know the difference. Too often advanced knowledge of technology is prized over other soft skills that are needed to be a good Senior Admin. Junior Admins have the luxury of focus. They are generally given tasks or small scope projects and that is their focus to completion. They at this level need to focus on the technology, learning how it works, supporting it. Whereas Senior Admins are responsible for more complex in scope projects as well as multiple projects. Seniors need to be able to effectively project manage the projects they are working on as well as overseeing the work of JSAs. At the senior level a tipping point occurs. As a Junior >50% of your time is dealing directly with machines. As a Senior >50% of your time is spent dealing with people. You're also expected to keep up on technology, as you're a Subject Matter Expert, or you should know who to go to if you need SME input. You have to put on the tie and go before meetings either to request budgetary funds or to present an upgrades or changes to the environment that can affect the institution. Its the bigger picture.

On a closing note I will pass on some wisdom once imparted to me by a trusted and valued mentor. Over lunch one day he asked me what I wanted to do in the future. I said 'I want to do systems administration until I retire.' He nodded, then asked 'Do you want to be on call for the rest of your life?' He gave some time for that to sink in. He then asked 'What if UNIX isn't around in 10 years?' Wow, I never thought about that. I assumed it would be, but UNIX is 30 years old, it could go away. He then went on 'You won't be a junior Admin for the rest of your life. When you get promoted you'll have to do more non-technical things [ed. he's right you know] and you can only go so far as a SA.' That lunch stuck with me as you can tell from my previous comments. I'm at that point now in my career where I really don't want to be on call for the rest of my life. I've been thinking a lot lately about what to do besides UNIX. Some of it is job dissatisfaction, unlike farmers or carpernters, its hard to step back and see the fruit of my labor. I've also realized that while I've expanded my soft skills (communication, time management, etc.) I'm really narrowly focused with my hard skills. I need to expand my abilities.

Plus I think it is time for a change. To quote Duke Leto from Frank Herbert's work Dune 'Without change, something sleeps inside us...The sleeper must awaken.'