DevOps in the DoD: The Foundation

I know not with what weapons World War III will be fought, but World War IV will be fought with sticks and stones – Einstein

Perhaps we are beginning to see what weapons will be at play. Not tanks or heavy artillery, but streams of binary code promulgating across the electronic frontier of cyberspace; weaponized and invisible, carrying the signatures of nation states and other organizations out to undermine, destroy, or otherwise introduce mayhem into our Nation’s institutions and infrastructure.

Episode 1.

DevOps? What da’Hell is DevOps!? The crease in the major’s brow appeared like a bolt of lightning. The silent signal of confusion and disdain thundered across the desk in an instant.

She had just taken command of a Department of Defense (DoD) program worth several hundred million dollars, and at the heart of the program was a cyber defense system that promised to be the latest and greatest Defender of the DoD Kingdom. Hundreds of millions had been spent, and the project team was in chaos mode, doing their best to fix defects, document updates, run tests, and deploy the latest changes to production without blowing anything up. The major was in no mood for a marketing pitch.

I could feel my face contorting into that stupid look you get when you are at a loss for words. I had to say something fast, but my rehearsed pitch took a nosedive, crashed and burned at the slightest rattle. It was replaced with a storm of thoughts of past DevOps successes, all blazing fast forward with such speed that I couldn’t assemble any kind of remotely intelligent response.  

The pause was threatening to become uncomfortable, when I leaned forward and responded…

Stop. Rewind. I’m getting ahead of myself. This is meant to be the first Episode in a series, chronicling our DevOps journey within the bureaucratic titan, the Department of Defense (DoD). I literally can’t spoil the ending since it hasn’t played itself out yet, so let’s start at the beginning.

Our focus with this series will be to document our experience with applying DevOps principles and practices in the DoD environment, because if we can get it to work here, it can damn sure work anywhere. As a scruffy engineering startup, with over a decade of DoD experience between our team members, we’ve put aside our cynicism, massaged our scar tissue and press forward to make a difference.

So What Does DevOps Have to do with Our Nation’s Cyber Defense?

In a simple sense, DevOps is about delivering the highest quality product or service, in the shortest amount of time possible, without requiring heroics, death marches, or any other chaos-driven spectacles. In the context of cyber defense, this means getting defensive capabilities out to production as fast as possible, so our cyber forces can effectively outpace our enemies in cyberspace. It also means establishing a support system that allows for rapid change and enhancement, continuous learning, and smooth operation with the minimal amount of human intervention possible.

The cool thing about DevOps is that it’s not a framework, like ITIL or the Capability Maturity Model Integrated (CMMI), so it’s not Yet Another Framework to learn, and there’s no certificate program to drain your bank account. It’s more of a movement, with some core principles that will likely compliment whatever framework you’re using. It flies high above the weeds, and serves to establish some fundamental guiding principles, or as defined in the DevOps Handbook, The Three Ways:

  1. The Principles of Flow
  2. The Principles of Feedback
  3. The Principles of Continual Learning and Experimentation

I won’t go any further describing DevOps in general, rather my intention is to show how we’ve started to implement it in the DoD environment. If you are interested in learning more about DevOps, you can get a great, and brief, overview from DevOps illuminati, John Willis, in his blog post, What DevOps Means to Me.

In this first Episode, I figure it’s best to start at the beginning, and DevOps starts with Culture.

Starting with a Solid Foundation

Although this series is about the DoD, or at least how we are applying DevOps to our DoD projects, I won’t be spending much time on the DoD in this Episode. We have to start with ourselves, as a company:

DevOps starts at the foundation of the company, and must be woven seamlessly into the company’s fabric as a dominant force to influence its culture.

DevOps is like Punk Rock. It’s cool like that. You don’t listen to the Sex Pistols, buy some Doc Martens, and become Punk Rock – just like you don’t grab some tools, implement some processes, and become DevOps. That’s all important, but superficial. The root of DevOps begins with the company’s culture. And the tricky thing about culture is that it cannot be bought or artificially manufactured. It will not be found in the boardroom or in any company policy document. It will be found at the water cooler, or coffee pot. It will manifest itself in the daily habits and routines of team members. It is, inevitably, the culmination of many little things we do, as a company, to influence individual habits and behavior.

The rest of this Episode will describe how we’ve aspired to build and sustain a DevOps culture at Foxtrot Division.

Honesty in Purpose

A solid purpose certainly helps drive culture, as long as each team member can easily draw a connection between it, and their daily work. I’m not sure if this comes with experience, or what, but in recent years I’ve come to understand, and accept, the fact that everything is connected. Of course this is not a unique epiphany, but somehow over time it has become real to me, to the point that it’s as certain as the Law of Gravity.

Learn how to see.  Realize that everything connects to everything else. – Leonardo da Vinci

I heard about this PBS series, The Carrier, that documented life aboard the aircraft carrier, USS Nimitz. In one episode, the narrator interviewed random sailors, asking the question, so what do you do? The answers were amazing, and without fail, consistent. Take the dish washer. He hardly paused with his answer, which was along the lines of, I help keep a sanitary environment to feed our troops so they can stay healthy and fly missions to defeat our enemy. Wow. Now that is some awesome leadership!

Great leaders will skillfully connect the dots of individual effort, project purpose, and company purpose to empower every team member with a clear sense of worth.

A meaningful purpose makes it easy for us to always see the bigger picture, in the context of our own work. Not only do we have a company purpose, but we also define a purpose for each project. This helps ensure that everything aligns, that everything connects. If we get a job to paint houses, all of our team members will quickly recognize a disturbance in the Force. Things aren’t connecting. What da’Hell are we doing painting houses?

And so we must always maintain honesty in purpose, otherwise, our purpose will become little more than a marketing veneer.

Creating a Fearless Team

There are no ax murderers on your team. That is, team members are not out to purposefully sabotage or otherwise screw up. Unfortunately, we as humans seem pre-programmed to assign blame when something goes wrong, and companies are no different.

A good indicator of culture is how we, as a company, expect our project leads and other leaders to respond to those oh shit moments. You know, when defective code crashes the system, or a vital piece of hardware is not shipped to the production site. Does our response include assigning blame? Do we investigate with witch-hunt enthusiasm, to identify the culprit? Or do we avoid any blame whatsoever, and instead focus our enthusiasm on identifying the root cause, and learning how to prevent it from reoccurring?

We avoid assigning any blame whatsoever. Our position is that it’s not the team member’s fault – it is a failure of a process, a procedure, or a control. Misplaced optimism? I don’t think so. It took me a bit to get to this blameless approach, since most of the time it is human error, right? Someone jacked up the code, or forgot to ship the widget. Yes, technically true, but as we see it, assigning blame is really pointless.

We do not ask, why didn’t the team member follow the process, instead, we ask what obstacles prevented the team member from following the process?

Not only is this part of a DevOps culture, it’s a fundamental ingredient for effective continuous improvement. By empowering team members to embrace mistakes or failures as a natural occurrence, we erase any fear of reprisal, and foster a learning culture. Mistakes are seen as opportunities to learn how to get better, as a team and as a company. On the contrary, a culture of blame forces mistakes and failures underground, and rarely will any learning take place on a company-wide scale.

One of the best books I’ve read on creating a culture conducive to continuous improvement is Toyota Kata, by Mike Rother. Rother explains how Toyota’s world-renowned production system is based on a blameless approach to mistakes.

Create Core Values that You are Passionate About

Core values are another great influencer of culture – if done right. You have to be passionate about them, otherwise, they’ll likely become just another layer of marketing veneer.

At Foxtrot Division, we refer to our core values as our DNA, since they represent individual attributes that form the fabric of who we are.

  • we THINK
  • we SPEAK
  • we LEAD
  • we strive to BE GREAT at everything we do

From the DevOps perspective, the goal of our DNA is to institutionalize those sacred attributes that we believe will foster the individual skills necessary to effectively communicate, collaborate, and continuously learn.

Before we interview any applicants, we send them an applicant brief that describes our DNA. This way, before they are ever hired, they are exposed to our core beliefs and expectations. If values are to hold any weight, they must be a key consideration when interviewing and ultimately hiring applicants.

Much of our DNA was inspired by my time as a Unix Lab Tutor at the University of West Florida. I met a whole bunch of students, at all capability levels. The best part of this was that I began to see common attributes in the really great students.

Their ability to be great had nothing to do with their capability level at the time – it had everything to do with their commitment, and determination to figure out complex problems.

Jack Welch, former CEO of General Electric, referred to this concept as a person’s runway. So when I saw these particular students walking towards my desk, I knew it was going to be a hard problem, and more often than not, we both ended up learning something. Some of these students may actually be reading this, haha. You know who you are, and you’ve made a lasting impression. Thank you for that.

Other major influences that helped us create our DNA include:

  • The Cluetrain Manifesto – a fantastic book that slaps traditional command-and-control communication models in the face. Team members are the best marketing tool we have, and affording them a voice, and allowing them to speak freely and openly to clients is the true test of a winning culture.
  • The Leadership Engine – another great book by Noel M. Tichy that describes how to create a leadership culture and build great leaders at every level.
  • The Bell Leadership Institute – definitely the best leadership course I’ve attended so far. To be a great leader, we must understand ourselves, and most importantly, how others perceive us. Dr. Bell brings to bear a lifetime-worth of research in his leadership seminars.

Empathy

Did this just become a cheap romance novel? Empathy? Are we going to hug each other before and after every project meeting, and maybe in between meetings just because? No, it’s not the 70s anymore. In this context, think of empathy as our ability to connect with one another, and understand each others wants and needs and motivations.

We’re no strangers to empathy, since it plays a vital role in User Experience (UX). After getting into DevOps, it became clear that we needed to expand our definition of users. Many of us in the UX field know we must empathize with our end-users to create a successful product; however, what about all our team members involved in running the business, developing the products, and providing the services? The testers that test our code; the system administrators who set up the servers and the tools needed to maintain the system; the techs that stand watch at the Help Desk; the sales team that sells the product.

By extending our definition of users, we now apply UX to just about everything we do, including developing company policy, and implementing new tools. When creating company policies, we have three absolute requirements:

  • It must comply with all applicable laws and regulations
  • It must work for the owners of the policy
  • It must work for all team members impacted by the policy

A good example is timekeeping policy. Accountants are normally the owners of this policy. Too often these policies are written by these accountants, and supported by tools that help the accountants crunch the numbers. Unfortunately, us team members who have to use the timekeeping system are rarely considered and we’re left to fend for ourselves with what is inevitably a clunky, pain-in-the-ass system.

I’ve used timekeeping deliberately since it is a personal pet peeve of mine. As a case in point, I was working for a huge defense contractor that had an online timekeeping system. The JOB CODE field had a dropdown selection, but every option was some obscure accounting code, with no title or description. I’d been coding all day, and after about ten minutes of fruitless scrolling through job codes, I finally just picked one and went home. The next day I received an email from a manager in some distant branch asking why I was charging to her job code. Turned out that the code was for some project that had to do with delivering weapons systems to Iraq. Oops.

Again, it’s the culmination of many little things that serve to develop our culture. If we simply continue to adopt this tired-old model of business-as-usual, use legalese and defensive language to protect ourselves, and force shitty systems on our team members, then we will get the culture we deserve. If on the other hand, we empathize with our team members, trust that they are not ax murderers out to get us, and truly consider their perspective when developing policy and implementing tools, we move closer to building the culture necessary to successfully implement DevOps.

Making a Difference in the DoD

To us, introducing DevOps into our DoD projects is more than just an interesting experiment. There’s something much bigger at stake for us. We want to contribute to positive change, and help raise the bar for defense contractor performance, especially in the area of cyber defense. It is truly shocking to know how much taxpayer money is spent on inferior quality products and services. I have no doubt that many defense contractors would never survive in the private sector. Hmmm, you built me an application that looks like shit, behaves like an asshole, and crashes almost every month. You’re not controlling changes because it’s not in the scope of your contract?

For us, raising the bar starts with building a solid DevOps culture to empower high performance, at less cost and in less time. Most importantly, we can help get much-needed cyber defense capabilities deployed faster, more reliably, and with less manual love and attention required to sustain them.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s