Friday, January 9, 2009

So you think you want a job in Computer Security

This is my blatant attempt to re-direct any aspiring, up-and-coming security professionals into another line of work, for the sake of their own physical and mental health.

So, you think you want a job in Computer Security, eh? Are you sure? Have you been properly informed what the work and conditions are really like? Do you have visions of Hollywood movies where Cheetos-eating one-handed-typists are madly furying away any would-be "hackers" and think you "want a job like that"? Or have you just heard about large salaries and want to make some extra do-re-mi for another coat of white paint on your picket fence? Or maybe still, you're one of those who think the "enlightened" few computer professionals rise above to the pinnacle of computer security research or applications, and you want a piece of that intellectual satisfaction?

Regardless of why you have been considering a job in computer security (or maybe you landed into one and you're wondering "How did I get here?" and "Now what?"), it is extremely likely you're missing a bit of a reality check you could have used prior to now. Now for a dose in reality ...

  1. Perfect Security is not possible. It's not. It's depressing, I realize, but it's not. You may be surprised to find so many people working {Computer, Information, Network, System, Application, Software, Data, IT} {Security, Assurance, whatever} jobs who don't get that. I must admit that a former, more naiive version of myself once thought computer security was just getting some complicated recipe of hardware and software components just right. There's still a surprising number of "security professionals" out there who think that way. It's very depressing, but there's a very large "surface" to protect and it only takes a microscopic "chink" in your armor to lose everything. As a result, perfect security being not possible is the foundation to all other reasons why you should seriously re-consider your career aspirations.

  2. Most security work is really about making sure everyone else does their job "correctly". Correctness of systems is the real task at hand in a security job. Is it correct that a website of known sex offenders allows the general public to inject records of anyone they want labeled as such? Is it correct for a web server to execute arbitrary code if it is passed 1024 letter "A" characters? Is it correct that a user can click on a link and divulge intimate secrets to a total stranger because the page looks "normal" ? None of these are "correct" assuming even a smidge of common sense looking on afterwards. Yet they all have happened, and it was some security professional's job to deal with them. To put it simply, if everyone figured out how to design and implement systems "correctly" (assuming they know what is "correct" and what is "incorrect"), then security professionals would be out of a job, but thanks to #1 (perfect security is impossible), we're guaranteed to be picking up the poo poo flung by others from now until retirement, which means the following ...

  3. Security Response jobs suck. It may seem like CSI or something, but jobs that deal with responding to incidents suck. Except in high profile cases, computer forensics and true chain of custody techniques are not followed-- and if you want a computer forensics job, you'll probably have to work for a large government/public sector bureaucracy (and all the fun that goes with spending tax payers' dollars), which means you'll be primarily working on child pornography or drug trafficking cases and riding daily the fine line between public good and privacy infringements (warrantless wiretaps come to mind). My anecdotal observation is that very, very seldom do drug dealers and child porn traffickers actually employ decent computer security tactics; therefore, the job is lot less "CSI" and lot more mind-numbing "lather, rinse, repeat". From the words of someone I know who does this work: "I pretty much just push the big 'Go' button on EnCase [forensics software] and then show up at court explaining what it found." Not exactly the most intellectually stimulating work. The coolness factor wears off in the first 90 days, plus there's the joy of having convicted felons know who you are and that your work put them behind bars-- but not quite long enough, as they might still have a grudge against you when they get out. Even if you're lucky enough to not have a begrudging felon on your hands, there's the deep psychological torment that will slowly boil you alive if you are constantly exposed to the content of criminal minds. Your mileage may vary, but it probably won't be what you expect.

    For those who hope to work responding to computer intrusions, you should realize that very few organizations can afford to keep people on staff who perform only computer intrusion investigations. Most orgs just want to know what it will take to get things back to normal, because to do a full root cause analysis on a computer system that generates revenue, well, that likely means the org will have to forego revenue, at least long enough to take a forensic snapshot of all of the data. Very rarely (mainly just high profile cases), will an org be able to afford that. So the competition is tough. Not to mention that in many publicly traded companies, there is indemnification from not knowing exactly how an intrusion occurred. And there's even more stigma if the details are made public. So there's just no incentive for them to really find out all of the details. The 20,000 foot view is good enough (e.g. "vulnerability in a web server").

    And then there is an entirely different breed of "computer security professional": those who work on disaster recovery and business continuity planning and response. As you get engrossed in this sort of work, it tends to be less about "security" (critics: I realize "availability" is a tenet of the CIA Triad) and more about the daily employ of scare tactics to get organizations to fund remote data centers that are ready for the next apocalypse. The work is surprisingly more akin to "facilities" planning work: buildings, electric, plumbing. There is a "cyber" aspect to it, but it's mainly about funding the necessary equipment and then getting sysadmins to build it and test it out. That's project manager work; tedious, nanny-like, often political. It's not for people with short attention spans or high expectations.

  4. Security Operations jobs suck more. Security Ops is at the bottom of the security professionals' totem pole. Most of these jobs are just sysadmins or network admins who have been promoted an extra notch, maybe because of that shiny new industry cert that some trade rag said was "hot" and would result in a 15% salary increase. But all of the usual sysadmin/network admin griefs apply here and then some. It's an operations job, so you inherit all of the problematic decisions that the project planning and implementation people lopped over the fence at you. Very rarely do Security Ops people in an org get to influence the architecture of future deployments. And besides lightweight tweaks like patches or an occasional config change, very rarely do Security Ops folks get to do much to systems "in production", especially for "legacy systems" (what part of "legacy" isn't a euphemism?). For the most part, it's sit back and watch to see if a security failure occurs. I use the word "failure" with specific intention, because Security Operations folks have to constantly keep delicate China plates spinning atop poles, because each plate represents a certain security failure. As it is with spinning plates, it's often about deciding which failure is more acceptable, not about preventing all failures (see #1, again).

    In fact, there's an interesting twist: Security Ops managers or directors who experience a breach may find themselves losing their jobs on incompetence grounds. Going back to #1, this seems counter-intuitive. If we know perfect security is not possible, then we know security operations will experience a breach at some point (if we give them enough time). How, therefore, can you ever expect to be successful at a security operations job? When the shareholders want to know who was responsible for the unauthorized disclosure of thousands of company-crippling account records, the first person with the cross-hairs on their back is the person in charge of security operations. So, to survive at this game requires either company hopping before the inevitable breach occurs, OR, it requires politics (or black mail on somebody high up).

    Outsourced security operations is just a variation of this. If the contract includes full accountability, it's one and the same as what is described above. If it's a "we monitor your systems that you are accountable for" scenario, then you as an individual security operations employee of the contract firm may not get fired, per se, but your company may lose the contract renewal, which means if you allow #1 (above) to be true too many times, then you might find yourself out of a job there, too.

    The worst part about SecOps is that you'll either realize you've hit your Peter Principle with that job, in which case it's time to spend all of your free time on backyard barbecues and retirement planning (nothing necessarily wrong with that -- ignorance is bliss), OR, you'll want out immediately because everyone around you has hit their Peter Principle highest job and you want more.

  5. Security Planning jobs are set up to fail. Think about it: perfect security is not possible. So, even the most cerebral of security planners is going to deliver a work product that has flaws and holes. If you can convince yourself that's not depressing and continue on, maybe you can also be lucky enough to get into an organization whose culture thinks it is acceptable for people to deliver faulty products to a Security Operations group (#4 above)-- and that it is entirely the Operations' people's faults when it capsizes. Not to worry, though, you probably won't work for an organization that can afford a true security response group (#3 above -- it's probably just the Security Operations' people who get to handle the full response process to break up their mundane day), so nobody may know it was your fault. Besides, if you're dealing with a bunch of vendors' COTS (Commercial Off The Shelf) wares, there's not a whole lot of control for you to have, which begs the question why your organization even has a position for you in the first place. They probably could have just paid some consultant for a couple weeks, rather than have you permanently on staff.

    The other downsides are, of course, that you (like the Disaster Recovery & Business Continuity Planners) will also have to use scare tactics to implement draconian policies which probably won't actually amount to any real benefit, but some "power user" or Joe Software Developer will figure out he can circumvent them if he has two laptops and a flash drive (long personal anecdote story). If that doesn't work (or if you just want to cut to the chase), enter regulatory compliance into the equation: "Your project must do that stupid, expensive thing that results in no real added value because PCI says so!" It won't be a policy for something that 100% makes sense 100% of the time. Instead it will be something that makes life difficult for everyone (and everyone will love you for that), but is generally accepted by 3 out of 5 security professionals who also have no clue and are stuck in the dark ages (hence there are a lot self-perpetuating bad ideas out there, like firewalls and anti-virus). If you're an enlightened security strategist, you'll realize the futility of your job and want out, or you'll revert to also longing for weekend barbecues, vacations, and eventually retirement, all the while wondering if this is your Peter Principle job.

  6. Security vendors have to sell out. They sell out because they thrive on the perpetuation of problems, selling subscription services to deal with them. Scare tactics are used so frequently the vendors are numb-- finding themselves unaware they're even using them. Not to mention, there are so many security vendors out there for startups and small boutiques alike that most security professionals on the potentially-receiving side of their goods and services haven't even heard of them. Or maybe they have? The names all sound so familiar, like: Securify, Securification, EnGuardiam, Bastillification ... they all seem to make sense if you're still in that state of mind after having woken up from an afternoon nap's dream, otherwise they reek of a society with too many marketing departments and far too many copyrighted words and phrases. If the company is any good, they'll eventually be swallowed up by one of the bigger fish, like Big Yellow (Symantec), Big Red (McAfee), Big Blue (IBM), or one of the other blander colors (HP, Microsoft, Google, etc.). Only a few stand strong as boutiques, and if they do, they almost certainly have a large bank or government contract as a customer.

    Once you get a job at a security vendor, you'll probably be working as a developer who maintains a security product. And, as Gary McGraw has often pointed out, that's not about writing secure software, that's about writing security features into software. If you're not maintaining it, you'll be supporting it, which is the exact same as Security Operations (#4 above). You'll be the low level person who is stuck taking tickets, interpreting manuals (RTFM!), and talking to the Security Ops people at your customers' orgs. Fun times. Don't think for a second you'll go get a job at one of those big companies and fundamentally shake up their product lines and come out with cool new security-features-software that the Security Ops folks could really benefit from. These big companies get new ideas by buying the startups that create them; rarely does a lightbulb idea make its way into fruition. In fact, if you have such an epiphany and develop it as your brainchild into a security startup, rest assured that the bigger fish that swallows you up will succeed in turning your baby into yet-another-amalgamated product in their "enterprise suite" of products and services. It will lose its luster. They'll make the UI match the "portal" their customers already love to hate, but by then, you will have sold out and you can take your new nest egg with you into early retirement (weekend barbecues, here you come!).

    If you're not one of those, then you will really be a sellout-- either a sales rep or a sales engineer. If you are somebody who like repeating what you say and do, this is the job for you, because you'll repeat the same lowly power point slide deck that marketing (you remember-- the people who came up with that killer company name!) for every customer-- that is, all of the customers that let you in past the cold call. If you're the sales rep, remember to drag along your sales engineer to get you out of a sticky situation where you promise some security perpetual motion where it's just not possible. And if you're the sales engineer, try to remember the security perpetual motion is just not possible. It'll be hard to tell the customer that, though, since it will say otherwise in the power point slide deck that marketing provided. It's be right there in big red letters: "Secure", "Unbreakable", "Keeps all hackers out", etc., etc., etc.

  7. Pen Testers and Consultants have Commitment Issues. You can sell out, collect a paycheck, and position yourself in one of the jobs with the least amount of accountability and responsibility in the entire InfoSec space. The same is true for third party consultants, too. Any job where you are hired to come along and tell the hiring org where to put more bandaids falls into this category. Sure, there's a broad body of knowledge to comprehend ... but there are plenty of security vendors (see #6 above) who think they have a tool they can sell you so that you can point and click through your brief engagement with the hiring org, which begs the question: Why should they even hire you if an automated tool can give them their results? That's not true of all independent consultants and pen testers, though. Some of them do provide usefulness beyond that of a canned COTS tool. But they all suffer from the same problems as Security Planners (#5 above), only they probably had a prior job working directly for the org and saw how painful it was to stick around through the accountability phase after an incident. So now, they've learned their lesson: get in, get out, cash the check. They say: "Hey, it's a living." Are they the smartest security professionals around? Maybe. Do they have what it takes to do the other security jobs like Planning, Ops, and Incident Response? Maybe not.

  8. Exploit writers perpetuate the problem. All they do is sit on a chair all day in front of multiple computer screens (no doubt), and attempt to prove over and over again what academics have been saying since the 1970s. Yet there seems to be some economic sustainability, because otherwise the security vendors (#6 above) would have no way to sell you subscription services to access today's latest hack that a criminal otherwise might find on their own. But thanks to the vendor (and their handy, dandy exploit writer they have locked up somewhere with unlimited access to caffeine), we can all rest safely that the exploit code they just wrote won't be weaponized to prove #1 again (that happens all the time, actually), causing some poor Security Ops person (#4) to get sacked, while some Security Planner (#5) thinks "glad I'm on this side of the fence", and some Pen Tester (#7) thinks "I gotta download that into my pen testing tool for tomorrow's gig-- that way I know I'll find a hole and they'll hire me back next year".

  9. Security Educators either are paranoid or should be. If you're just contemplating a career in information or computer security for the first time, you probably aren't acquainted with any of the lovely people in this category, mainly because the good ones are expensive. Typically, it's only existing security professionals that get to experience security educators, because their employers realize that it's important to keep them up to date with information-- primarily thanks to exploit writers (#8) who keep the litany coming. The principles of security rarely change; only the scenery changes (and the exploit writers change scenery like the masters paint in oil).

    Educators fall into one of two categories: 1) they suck because they've been out of the game for so long (if they were ever in it at all), or 2) they're spot on, but they don't want you to know what you're reading now because you may consider a career change and that's one less pupil, one less paycheck for them. If they're on top of their game, they're paranoid. They have trust issues with everything and everyone. They can't stay away from the topic, so they're very well-versed in what has happened as well as the current goings-on in the field of security, but they have worse commitment issues than Pen Testers and Consultants (#7). They have the ability to scare you, but not in the same way as the security vendors (#6) and security planners (#5); you'll be able to tell that they don't want anything in return-- it's almost a relief for them to share the information they know with someone. Sometimes a vendor sneaks in and pretends to be an educator. Beware of that; though the way to spot them is their horror stories will result in an emotion to buy a product or service. You won't come out having learned anything other than their products solve a niche need.

    Becoming a security educator isn't an easy task; it typically means you were an educator of some other specialty domain and then learned how to teach security (which usually doesn't work as well as someone who has lived it), or you lived it yourself through one of the other job types and have educated yourself beyond the level of ordinary practitioners. If you're already in a security career and find yourself disheartened by the lacking options around you (because you've realized that it isn't the glamorous field you once thought), but find that you have an amazing affinity towards learning all that you can, this might be a saving grace that will prevent you from leaving everything you've learned behind and taking up a job as a dairy farmer (or some other similar job that will not require you to touch a computer). There's also the potential for life as an academic, where you can infiltrate inspire open minds that have yet to be corrupted by corporate ways.

  10. Security Media don't really exist. There are like 4 or 5 real "computer security reporters" in official media outlets. Anyone wanting to aspire to be them would have nearly as good of odds at becoming a professional athlete-- and that pays better. For all intents and purposes, they're either vanilla columnists whose writing glares that they don't understand the technical underpinnings of the subject of their writing, OR, they're paid bloggers.

  11. And Security Bloggers are the worst above all. (Present company included.) They know some or all of the above and chronicle it where they can, thinking that just collecting their thoughts in some digital pamphlet will change things. In order to be a security blogger of any real significance, you have to be known among the security community. For most, that means affiliation with a brand, product, or service. For a very elite few (the Schneiers out there), that means being one of the first to do so, calling everyone out for who they are, and taking as many opportunities to spout off in normal press/media as they'll allow (e.g. Schneier's a self-proclaimed "media slut"). For the rest of us, this may just be an attempt to alleviate the pressure of painful security information in our brains-- a pressure-release valve.
Do you still think you want a job in computer or information (IT) security? If your sole motivation is a paycheck, even if it means beating your head against the wall while trying to solve unsolveable problems, then this may be a career choice for you. If you can survive without gratitude for a job well done (because when these security professionals are actually successful, by dumb luck or otherwise, they largely go unrecognized and unthanked), then you may have a chance.

If you hope to change the world with your career, may I suggest a rewarding opportunity teaching high school math or science in a public school system? The pay is for shite, and there will be harder days than being a security professional, but your pupils will be grateful for your job well done later in life-- even if they don't manage to get around to tell you. Besides, everyone knows Americans spend what they make-- just learn to make ends meet on a teacher's salary.

[My general apologies for starting off 2009 with a lump that is hard to swallow.]