I Propose a Truce

I’ve been a software developer for 20 years, and for about 15 of those years, I’ve worked in secure government facilities or with other large, security-conscious organizations. I used to have to deposit my cell phone in a locker, walk through a metal detector, and swipe my badge multiple times just to get to my workstation. Over the years, most of the developers I’ve worked with have been hard workers who just wanted to build good software. They love solving interesting problems, and working with cutting edge technologies to build applications that allow their users to focus more on what they need to do and worry less about how they’ll do it. But throughout most of my career, my development objectives have invariably run up against the dreaded roadblock of cybersecurity.

• “No, this library hasn’t been officially vetted by security, but it’s the most widely used message broker on the market. It’s fine.”
• “I can’t give you a finalized architecture diagram until you let me install this software so that I can prove that my architecture will work.”
• “Why do I have to talk to 7 different people just to get a port opened in the firewall?”

I’ve worked with many solid developers who described the places we worked as simply, “The place where progress goes to die.” It can definitely be disheartening, to say the least. But still, you have to ask yourself, why does all this security red tape exist? And hopefully, in most cases, there’s a good answer.

If you’re building a website for your daughter’s Girl Scout troop to sell more cookies, then sure, there probably isn’t a need for a ton of security. Not ZERO security, but you don’t have to code Fort Knox. On the other hand, if you work for a bank, the power company, or the Department of Defense, then obviously there should be more security in place, because the penalties for failure are so much higher. If you wake up one morning to find that your slap-dash code allowed a hacker to post porn ads on your daughter’s Girl Scout site, well yeah, that sucks. You may have some explaining to do to some angry parents. But if you come in to work to find that a foreign adversary compromised your server full of classified mission briefs, and as a result, last night’s “door-kicking” raid put one of our soldiers in a wheelchair, you’ll be explaining yourself to more than your daughter’s Troop Leader.

Ok, maybe that’s a little extreme. Let’s take it down a notch. Clearly, in matters of life and death, the utmost in security protection is warranted. But that’s not the average situation. Sometimes it feels like cyber is just issuing blanket rejections to your requests for no good reason. Why is that? Well, one (overly-simplistic) reason might be:

1. Security breaches happen
2. In the aftermath, there’s a lot of finger pointing

We live in a litigious society, and when the time comes to assign blame, there is a long tradition of looking for the scapegoat. If you are the developer that wrote the code that introduced the vulnerability, well you’re a great candidate to take the fall. But, if you work in an organization that has a full-fledged security team in place, it might be the cyber analysts on the front-lines defending their actions leading up to the breach. “Hey, I’m just a developer. You hired me to write the code to make it work, not to keep it secure.” That is often the prevailing climate in some of these institutions. And if you were a security professional in one of these organizations, wouldn’t you tend toward saying “No” first, rather than potentially hanging yourself out to dry?

Consider the 2017 data breach at Equifax. Somebody failed to patch a critical Apache vulnerability, resulting in the theft of the personal data of 143 million people. It’s hard to know how many individual security worker-bees were let go, but that one went all the way to the top, ending in the forced retirement of the CIO and CISO. If I were an incoming cyber analyst that was shown that brief on day one, you better believe that on two I’d be busting out that big “DENIED” stamp and putting it to good use.

I said at the outset that I’m a software developer. In some ways, that’s the “team I play for.” And I have to admit that I’ve spent my fair share of time griping about all the security overhead that keeps me from doing my best work. But in my case, it’s not quite so simple. The DoD contracts that I worked on required technical staff to have some sort of bonafide security certificate to even lay hands on a keyboard in those facilities. And my manager at the time, in an effort to maintain a competitive advantage in contract negotiations, insisted that I study for and sit the CISSP exam. I didn’t want it. It meant a lot of work studying an almost 2,000-page book filled with all sorts of security minutiae, in which I was only mildly interested. I never wanted an actual security job. I knew that even having those five letters attached to my name might put me in front of the bright lights of the Whodunit Squad in the days after a security incident. But I did as I was told. I studied hard, sat the exam, and earned my little gold CISSP tie tack. And then I proceeded to keep my head down and act like just another developer who doesn’t have a certificate that would indicate that maybe he should’ve known better.

But, the fact is, I do have that certificate. I guess technically that means you could also call me a “Security Professional.” Heck, it’s right there in the last two letters of that credential I hold. So I do sort of see myself as a card-carrying member of both of these warring tribes. I really did wade through those 2,000 pages. I saw the sheer volume of information that a supposed “certified security professional” was supposed to know (and guard against). Occasionally, when I am commiserating with my development team over the latest cyber rejection or delay, I find myself saying things like, “Well, but let’s try to put ourselves in their shoes…”

Lest you surmise, from that description of the Equifax breach, that it’s only the security folks that get sacked in the aftermath, there are plenty of published security incidents that can be tied directly to the developer. Often it’s due to negligence (Hello, hard-coded credentials!), especially when tied with efforts to shortcut or bypass known security controls. And while, for legal reasons, it’s uncommon for companies to publish lists of individual personnel who were disciplined or let go after these incidents, someone is eventually going to run a git blame on the code. If your name comes up, you’ll likely find that command’s name to be uncomfortably literal.

So, what’s the solution? Well for starters, you could look for a different job. Let’s face it, software development and security can be exciting, challenging, and highly rewarding careers, but they’re not without risk. There are safer alternatives out there.

Another solution: the Bugout Bag. I used to work with a systems administrator that managed all of our Linux servers. I asked him one day what he would do if he messed up and lost the whole database, with no possibility of recovery. He said, without missing a beat, “I’d just go home and not come back. Probably move.” I laughed and waited for him to crack a similar smile. He didn’t. He meant it. He had thought about it and that was his strategy. This was a middle aged guy with a wife and two kids. And his solution to the security perils of his job was to keep a bag packed, stuff cash in his mattress, and not sign any long term leases (so to speak). That’s bananas! Who wants to live like that? Not me.

So, what about a third option? What if we tried to work with each other, instead of glaring at each other across the security breach demilitarized zone? We don’t need to always assume the worst of the other team. “Oh man, that guy’s gonna get me fired.” “Dude, if they ever got off their butts and let us DO anything, then maybe we’d meet our deadlines!” Sometimes it seems like we’re just gladiators competing for the coveted prize of “least responsibility.”

Part of the problem may be that we tend to introduce security at exactly the wrong time. If the first time the cybersecurity team gets a good look at your application is two weeks before you’re planning to deploy it, you’re setting everybody up for a bad time. By then, the developer has deadlines to meet, management is expecting delivery, and decisions that were easy to change six months ago are now buried under 40,000 lines of code. Cyber finds a problem, development sees a roadblock, and everybody retreats to their respective corners.

What if instead we brought security into the conversation while we were still drawing boxes and arrows on a whiteboard? If Cyber tells me early that the clever new technology I want to use is going to be a non-starter, I might grumble about it, but at least I haven’t spent three months building around it. Better yet, maybe instead of telling me “No,” they can explain the risk they’re worried about. Now we have an engineering problem to solve, and engineers tend to be pretty good at those.

Maybe we can isolate the component. Maybe we can change the authentication mechanism, restrict the firewall rules, encrypt something we weren’t encrypting before, or redesign the data flow. Maybe the answer really is “No, you absolutely cannot do that.” Fine. Sometimes it is. But if we have that conversation early enough, we still have time to find another way. In fact, let’s have those conversations early and often. Regular security reviews can be helpful checkpoints instead of crushing roadblocks.

None of these things eliminate security requirements. That isn’t the point. They make the requirements visible earlier, when they’re cheaper and easier to deal with.

There’s an organizational component to this too. If development is rewarded entirely for delivering features quickly while Cybersecurity is rewarded entirely for preventing risk, we probably shouldn’t be surprised when they behave like opposing teams. One group is stepping on the accelerator while the other is standing on the brake. Maybe the better goal is one that both groups can share: deliver useful software as quickly as we reasonably can, at a level of risk the organization is willing to accept.

Hey developers, how about we try to do less griping when security policy invariably puts the brakes on our otherwise flawless sprint cadence? There might just be a good reason for the detour, and it might just save our hides. Better yet, let’s invite those security folks along for the ride at the get go.

And security folks, why don’t we maybe put that “DENIED” stamp back in its quick-draw holster and tell us what you’re actually worried about? Give us the constraint and let’s see if we can solve it together. Developers spend their careers finding creative ways around technical obstacles. Let’s point that instinct at the security problem instead of making security itself the obstacle.

Mistakes happen on both sides of the dividing line. A little grace for our counterparts—and a little more communication before the deadline is breathing down our necks—could allow us both to trade our bugout bag for a carryon filled with swimwear and sunscreen, so we can go kick back on a beach, knowing that our teammates are keeping an eye on things back home.

Bottom line: Life is hard. Work is hard. If we can work to shift the Venn diagram a bit, so that the development and cybersecurity teams are both in the overlapping intersection, well, that’s a team I’d sign up for. Because, in the end, there are only two types of people in the workplace:

1. A group of highly-fallible, flawed individuals who nevertheless try to do the right thing, play nice with others, and own the mistakes they make
2. All the other turds out there

Let’s not be turds. 💩