Circa 2003. Joy 3-2581, fresh out of grad school, begins his career as a software engineer in the bowels of a four-story brick building on the edge of Naval Air Station, Pensacola. Writing code was his passion, and the added bonus of contributing to our Nation’s cyber defense was a privilege not overlooked at the time, although the true extent of that responsibility was far from realized.
Essentially a one-man team, Joy 3-2581 buried himself in code, and new features he dreamed up, and pretty animated charts that visualized anomalies in network traffic at critical points in the infrastructure. Far from his mind was the security of his own application, after all, the majority of his time was spent getting capabilities out to our cyber warfighter so they could better defend our Nation in cyberspace.
And then it happened.
Three-weeks worth of code disappeared in an instant, when the nucleus of Joy 3-2581’s creativity, his fruit-bearing laptop, crashed hard.
Uploading code to an external version control repository, what? Okay. Noted. Version control repo configured and activated. That won’t happen again. Fortunately for Joy 3-2581, software development was kind of a hobby-shop within the organization, so no hammers were dropped and the three-week loss of productivity went entirely unnoticed.
Eventually Joy 3-2581 got some help. Enter Crudeness 3-3240. Another fresh grad, Crudeness 3-3240 casually immersed himself in the project, and contributed code every so often, between MySpace postings.
And then it happened.
Users were greeted with the infamous error page that stated, Application Unavailable.
Phones started ringing. A wave of emails assaulted our inbox. The application was offline. Donning their detective fedoras, the developers checked the usual suspects. No leads. Logs were sparse. Error logging, what? Okay, resort to good-old police work. Hit the streets, or the code in this case. Joy 3-2581 began walking through every new line of code that was introduced in the past few months. Sure enough, twenty-three hours in to the investigation, the fatal line of code was identified: System.exit(). Yeah, this is like the kill switch for an application. Like any prudent developer, Crudeness 3-3240 was predicting errors that may occur during the application’s operation; however, instead of recovering gracefully from the error, Crudeness 3-3240 simply added the line, System.exit(), so essentially if a user made a mistake, the application would shutdown completely. In addition to error logging, two things became abundantly clear. Changes needed to be controlled, and Code Reviews are a good idea.
Years had been devoted to developing this cyber defense application, to defend our network infrastructure. Ironically, the system was never successfully attacked or otherwise compromised by a malicious actor.
In every case, losses of productivity, and system availability, were the result of internal mistakes, oversights, and other events resulting from internal causes.
Fast forward, many years later. Joy 3-2581 continued to develop systems, and eventually assumed leadership roles for several cyber defense projects. Security was paramount to project stakeholders, or so it seemed on paper. Throughout these years, Joy 3-2581 collected much paper, and sunk himself into the many rabbit holes of Department of Defense (DoD) cyber security policy, starting with the DoD Information Assurance Certification and Accreditation Process (DIACAP), to most recently, the Risk Management Framework (RMF).
With piles of DoD Instructions and Directives, and National Institute of Standards and Technology (NIST) Special Publications, spread across his office floor, Joy 3-2581 studied the interconnections of all the strategies, rationales, rules, and recommendations, and was at risk of becoming a mindless drone, a frankenstein agent for security. Pragmatism was lost. Strict adherence to policy was the marching order. At one point, Joy 3-2581 calculated that there were up to seven thousand individual security requirements that applied to his system. If he could implement all seven thousand requirements, the system would be secure! No one could argue.
Joy 3-2581 initiated Operation Boiling Point. A three-month operation during which the project team would focus tremendous effort on cataloguing every security requirement, and the current status — was it implemented? If not, what needed to be done to get it implemented? It was the ubiquitous, fall back and re-group strategy. The project team made much progress, but soon found itself overwhelmed with fighting current fires and responding to new stakeholders needs, and focus was lost. The project just could not catch up with the backlog of security requirements.
In between all the fire-fighting, Joy 3-2581 and his team struggled to implement antivirus technologies and two-factor authentication and every other technical control to prevent bad actors from comprising the system. Ironically, the system was being compromised over and over, although in a much more subtle manner than an outright attack. Changes would be made to the system, and the system would appear to be functioning correctly, although beneath the veneer of normal operation, it was actually critically ill. Think of a system that is meant to capture, and log, every attempt to access your network. If, after a change is made, the system is only logging 40% of the access attempts, that’s a problem. And this is not necessarily something that is immediately noticed. It is a subtle, but critical issue. Okay, change needs to be controlled, as in TESTED prior to release. Now this wasn’t the only problem. Remember, bad things happened over and over. Bad things, as in, those charged with our Nation’s defense could not access the tools they needed to defend our networks. What we, in the cyber security industry, call loss of Availability. Throughout Operation Boiling Point, Joy 3-2581 was under the illusion that a disciplined approach to implementing all security requirements would ensure the system would be secure.
Over and over, losses of Availability occurred due to lack of testing, lack of planning, and lack of change control.
It wasn’t the Russians. It wasn’t the Chinese. It was internal lack of engineering discipline. Discipline in the form of THINKING and PLANNING, not checking off boxes in the laundry list of security requirements. As Eisenhower said, Plans are nothing; planning is everything.
Joy 3-2581 had an epiphany after a seemingly routine meeting with Pragmatic 5-2550, the Authorizing Official (AO) for a major DoD agency. Like the Supreme Court, the AO is the final authority. In this context, the AO ensured all systems were secure, and had the final say in what security requirements could be waived, and whether or not the system could be deployed on the agency’s network. Up until that point, nearly every security advisor involved in the project was adamant about checking the boxes and ensuring all security requirements were properly implemented, and if not feasible, the proper forms for risk acceptance were completed and filed. Then Joy 3-2581 had his meeting with the AO, and years of frustration poured out onto the conference room table, not in the form of an eruption, but more of a respectful surge of hopelessness.
The process, as is, nearly ensures we are set up to fail from a security perspective. Risk, in terms of likelihood and impact, rarely enters the equation. We are simply faced with an encyclopedic list of security requirements to implement.
The AO, in a few memorable words, responded, I know, you’re just answering the mail. Tell me, what are the real risks I need to be concerned about? Yes, answering the mail. In plain terms, this means nose down, filling out forms to satisfy the bureaucratic machine. At that point, the dots were connected for Joy 3-2581. Here was a senior executive who understood the innate inefficiencies of the process, was grounded in reality, and was looking for some real intel and willing to have a down-to-earth discussion.
From that point on, Joy 3-2581 adopted a more realistic perspective on cyber security. One grounded in the piles of paper on his office floor, tempered by reality. Now, without a doubt, those piles of paper are an invaluable source of knowledge. If you read, and understand the underlying intent of the documents that define the Risk Management Framework (RMF), the Security and Technical Implementation Guides (STIGs) published by the Defense Information Systems Agency, and Industry best-practices such as OWASP Top Ten and Sans Top Twenty, you will be armed with a solid foundation to understand the full scope of cyber security. The next, and most important step, is understanding how to apply that understanding to build and sustain an effective defensive posture for your system.
What Joy 3-2581 learned over the years, is that in his case, the most likely threat was from the inside, and was not the result of malicious action. It was due to lack of sound engineering process. Lack of change control. Lack of thorough testing. The usual suspects that have bled engineering projects since the beginning of time. The encyclopedic volumes of security requirements, coupled with the predominant fear of attack, had led to a focus away from disciplined engineering process, to preventing deliberate attacks. Unfortunately, it seems the majority of costs, in the forms of loss of availability, increased financial burdens, and schedule delays, are often the result of oversights and other non-malicious events.
The Risk Management Framework (RMF) is definitely setting the course for improvement. At its core, RMF is all about a pragmatic approach. That is, yeah, we have over four hundred security requirements that theoretically apply; however, you need to THINK about these requirements in the context of your system, and apply requirements based on the real risks to the system.
Nothing in this post is meant to downgrade the importance of defending our systems against malicious attack.
DoD systems are poked, prodded, and tested over a million times a day by those who are out to do us harm.
Yes, that is an accurate metric. It’s like a million strangers jiggling your door handles and pushing on your windows, every day. Cheers to all our cyber warfighters, engineers, analysts, and professionals who give it their best to thwart these attacks. We must not; however, lose sight of the clear and present threat directly in front of us — ourselves.
Think of a project as a battleship. Many teams operating in many different compartments, all working towards a single objective – defend our Nation. At its birth, it is in dry dock, where its foundation takes form and prepares for battle in a production environment.
If we overlook certain foundational compartments, such as change control, requirements management, test and verification, these compartments will be plagued with cracks and gaps, and once in production, will immediately take on water.
If everyone is staring at the sky, looking to neutralize the next threat, which it can assuredly do based on the massive investment in its armament, it is easy to overlook the fact that the ship is taking on water, from below. Yes, the ship will be effective in defending itself from attack, but will be handicapped by the significant drag imposed by a weak foundation. It’s overall value to the war-fighting effort is diminished.
Although this might read as a work of fiction, it’s mostly based on real events (if you hadn’t guessed that already). To my friends and colleagues in the industry, thank you for the inspiration, and the example you’ve set for charging forward, in the midst of crushing bureaucracy and seemingly insurmountable obstacles.
To those leaders, who have the vision and the commitment to defend our Nation, and the willingness to engage in spirited debate and make decisions based on real intel (not the path of least resistance)…I’m proud that you represent my Government.
Let us continue to improve, see beyond the checkboxes, and thwart our enemies, both external and internal.