DevOps in the DoD: Integrating SECOPS

Episode 2.

Circa 2007. I was a few months in to a new software project for the DoD. We were a team of two, working for a large defense contractor, and racing the clock to get the app released. We were highly autonomous, nestled into a lamplit corner office outside of Falls Church, Virginia. Occasionally our manager would fly by and ask if we needed anything. Nope. Had everything we needed. Coffee. Cigarettes. Rock-and-Roll. Check.

We’d been shielded from nearly all management meetings with the DoD customer, until inevitably their discussions turned to obtaining an Authority to Operate (ATO) for our application. And so, with the next management fly by, I was introduced to the DoD’s program to certify and accredit systems that will operate on its networks. Our app would go nowhere without an ATO.

In the first fews hours I spent researching the certification and accreditation (C&A) process, I quickly realized the sheer enormity of it. Time to call in back up. I crafted an email to the Software Engineering division of our company, and asked for guidance on getting software applications approved for operation on the DoD network. I asked about this thing called a STIG, which I’d come to learn was a Security Technical Implementation Guide (STIG) that provided nuts-and-bolts security requirements for software applications, among other things.

I spent the rest of the evening cramming, sifting through DoD instructions, message forums, and the like. The next day I was elated to find a response from the Software Engineering division. After reading the first few words in the response, I knew backup was not on the way:

“What’s a STIG?”

I sat there, absorbing the fact that this major defense contractor had no documentation, insight, or any other guidance with regard to developing secure software for the DoD. The piles of DoD instructions and other security documents I’d been harvesting loomed on my desk like a weapon of mass destruction, targeting our app. Well, now we needed a fourth thing. Coffee. Cigarettes. Rock-and-Roll. Heroics.

Make Security Requirements Visible

Security requirements in the DoD can be like a distant freight train, unseen over the horizon, but its arrival is inevitable and, to the unprepared, potentially disastrous. Many of us have been there, swept up in the chaos as compliance activities start to show up on the schedule. Whether our team is responsible for compliance activities or not, the simple fact is that security must be fully integrated into the project on Day 1, before the freight train arrives. It seems that all too often our security teams are pushed into a bubble of autonomy, and kept busy creating and pushing paperwork to prepare for system authorization activities, while engineers busily stitch together components and develop a functional system. The compliance checks come too late, well after key design, development and integration activities have taken place.

A common phrase used in DevOps is Shift Left. That means, we shift certain project activities to an earlier point in the system development life cycle. In this case, we need to shift compliance activities to the earliest point possible in the life cycle, and the first step to doing so is recognizing what the security requirements are, and most importantly, educating all project members up-front.

In the DoD, there are three primary sources of security requirements:

  • Defense Information Systems Agency (DISA), Security Technical Implementation Guides (STIGs)
  • Information Assurance Vulnerability Management (IAVM) system
  • NIST 800-53 Security Controls (assuming the organization has transitioned to the Risk Management Framework (RMF))

What STIGs Apply?

Early on, we work with the cyber security team to identify what STIGs will apply to the system. Once we know this, we can get the applicable STIG checklist and incorporate it into our test and verification process. Any engineer responsible for a system component (i.e. a router, a switch, an appliance) is given the checklist and becomes responsible for verifying that the component is properly configured in accordance with the STIG.

Warning: Exposing your team members to the STIGs may result in verbal lashings, moans and groans, and hints of revolution. Suck it up. The learning benefit is tremendous, as a team member can begin to connect nuts-and-bolts security requirements to the actual system they are attempting to secure. Besides, if you have the right team (see Episode 1), insurrection will subside quickly, if it arises at all.

What Vulnerabilities are Present?

The DoD’s Information Assurance Vulnerability Management (IAVM) system is the authoritative source for announcing new vulnerabilities, and is largely based on the National Vulnerability Database (NVD). Tenable’s Nessus is a fantastic tool that will scan systems and determine if the system is susceptible to any vulnerabilities announced via the IAVM system. Nessus becomes another tool in the engineer’s toolkit to conduct security verification. Each engineer is trained on using Nessus, and incorporates Nessus scans into all test activities.

What Security Controls Apply?

Under the Risk Management Framework (RMF), NIST 800-53 provides the authoritative source for security controls that must be applied to the system. Of all the sources of security requirements, this is the most unwieldy and difficult to clearly hand off to an engineer. The reason being, NIST 800-53 includes a wide-ranging set of security requirements, including administrative type requirements (i.e. the project must have a change control process in place). Many of the administrative-type requirements would not apply to any specific system component, but rather, the overall project and/or organization.

We’re working on a repeatable process to distribute NIST 800-53 requirements to individual engineers, without burdening them with the security controls that don’t apply to their specific system component. With over 1400 security controls that may apply to the system, we definitely want to parse out those that don’t apply to a particular engineer’s area of responsibility.

Setting the Tone

We know the freight train is coming. It’s not enough to simply round up the project team for a kickoff meeting, and dish out long lists of security requirements to individual team members. The DoD’s compliance process is massive, painful, and prone to circumvention. As written in the Rugged Handbook:

In many organizations, the complexity of the current approach to security is overwhelming, and they resort to building a compliance based program. While their intended goal is to improve security, these programs often result in a “race for the bottom” where the organization is motivated to perform the minimal amount to achieve compliance, and certifiers are motivated to “check the box” for the lowest price.

We do not check boxes. We secure systems. This is our mantra, and this must be the perspective of all project team members. It takes the project lead to set the tone. It is okay to be non-compliant. We do not expect to be 100% compliant. Ever. Our mission is to secure systems, not check boxes. We must recognize what the threats are, what the vulnerabilities are, and prioritize. And most importantly, every decision you make, you must be able to defend. In other words, we have zero tolerance for bullshit.

We must resist the urge to mark the “Compliant” checkbox, and instead have the courage and conviction to be brutally honest, true to our duty as cyber security professionals, and choose “Non-Compliant” when there is insufficient evidence to choose otherwise.

Of course, this is difficult, even controversial, in an environment that is obsessive about compliance, especially when the word on the street is, You must have zero Category I findings in order to get authorized. (Category I findings represent vulnerabilities with the highest severity level).The takeaway here is that project leadership must establish the right project culture up-front, build trust within the team, and then fight the political and management battles as best he or she can, outside of the trenches.

To be honest, every time I’ve met with the authorizing officials who decide whether or not a system can be connected to the network, they’ve all had a very pragmatic attitude, and seek, more than anything else, to understand the true risk, not how many checkboxes are marked compliant. It’s all the gatekeepers in between the project and these officials that carry the hammers, and tend to focus on the numbers.

Make Security Issues Visible

Once a system has been tested, we will likely have security findings that need to be fixed. Often times we would have a dedicated cyber security team whose function was to coddle the centralized vulnerability management system – this is where all security findings are officially tracked. They would check the boxes, update plan of action and milestones, and monitor for new vulnerabilities that impacted our system. They essentially worked in a silo, managing this mysterious system that no one else on the team had access to.

We had to make these security issues visible to the entire team. Compartmentalizing cyber security information and vulnerabilities to a single individual, and a single, inaccessible system, was crippling us. Sure, we’d have meetings to discuss status and exchange information so the security team could feed the vulnerability management system, but you know how meetings go. It’s like drawing pictures in the sand. The knowledge gained is fleeting.

A key principle of DevOps is flow. That is, understanding how work flows through the project, and more importantly, understanding what the work is at any point in time. With this in mind, we decided to treat vulnerabilities, non-compliant controls, and other security findings just like we did any other issue. We would create a ticket, and as soon as we did that, our issue management process would take over. We didn’t have to create any new process, or stand up yet another tool. A vulnerability, just like any other issue, became part of our work backlog, aka technical debt. It would be prioritized, assigned, updated, and resolved like any other issue.

The cool thing was that our cyber security team need only log in to the ticket system to review the current progress being made, and they could inject their own comments, clarifications, and/or recommendations. Additionally, the entire team could, at any point in time, see what security issues existed, who was working them, and what the status was.

We use Intelink for our ticket system. It’s a fantastic, secure platform available for all DoD projects. For me, Intelink has been one of the most valuable tools as a project lead in the DoD, and I’ll definitely be referencing it again in future episodes. It uses Microsoft SharePoint, and the best thing about it is that project teams can have full control of their site. This is the critical. We often need to tweak our ticket system and other tools as we’re continuously experimenting and learning what works. We simply cannot do this on the more restrictive platforms (i.e. you can upload files but that’s about it).

Being a DevOps Change Agent

The Risk Management Framework (RMF) is here to stay, at least for now, and we must use it in a way that focuses on risk, not compliance. Otherwise, we’ll end up exactly where we were prior – checking boxes, pencil-whipping forms, and ultimately facilitating the illusion that our systems are secure (see The Illusion of Cyber Security).

To summarize our security integration efforts through the lens of DevOps:

  • We immediately expose all team members to security requirements, at the same time all other requirements become known
  • We build automated test cases that include security compliance checks
  • We build configuration scripts to automate system configuration, to include security hardening
  • We create a fearless project team that will be brutally honest when checking the compliance boxes
  • We make security findings visible to the entire team, just like any other bug, defect, or issue, so at any point in time, we know how much work is in progress
  • We focus on risk, not compliance

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s