About two years ago my team and I were white-boarding some design concepts in a nondescript building in Falls Church, Virginia. The designs were for a new, modular cyber defense system that would replace an aging system nearly a decade old (we’ll call it Medley 2-5491). Medley 2-5491 was pretty complex, and sat at the perimeter of a huge military network. Huge as in Internet-scale huge. Huge as in over seventy sites to which it was deployed. For as long as I could remember, we were tasked with maintaining Medley 2-5491, as its fans whirred loudly, and its disks spun, incessantly churning data and looking for bad stuff on the network. We lovingly referred to Medley 2-5491 as the Christmas Tree. It had all shapes and sizes of blinking switches, routers, and proprietary vendor appliances that did this and that to detect bad things on the network.
It was a chore to maintain Medley 2-5491. That’s an understatement. It was an epic, ongoing roundtrip journey to the apex of Mount Everest. Sorry, that’s a hyperbole and probably insulting to those who have actually reached the apex. At any rate, we had to ask ourselves, do we buy more of the same, and simply replace Medley 2-5491’s components with the latest-and-greatest vendor appliances, or do we divert to some outside the box thinking? Well, the erratic squeak of a dry erase pen across the whiteboard sealed Medley 2-5491’s fate. It was less of a squeak and more of a barbaric roar.
Our cyber defense system design would be modular, built with commodity hardware and open source.
We were driven by two factors. Cost and Flexibility. The Government was viciously slashing budgets at the time, and our cyber Warfighters needed more flexibility in the tools used to defend us in cyberspace. Trying to add capabilities to proprietary vendor appliances at over seventy sites across the globe, was difficult if not impossible. We were inevitably stuck with what we had, or faced with going out, buying new appliances, and deploying them to already power and space-starved data centers. Costs would be extraordinary. We had the benefit of working with a great team of Government clients, and once we did our cost analysis, the path forward was clear. And that cost analysis? We compared the cost of buying more of the same proprietary appliances, versus a commodity-based system that would provide the same capabilities.
A rough cost analysis indicated a savings of $8 million to the taxpayer by going with a modular, commodity-based hardware design.
So with a whole lot of coffee, cigarettes, and rock-and-roll, we got to work on Modularity 2-5492, Medley 2-5491’s replacement. What previously could only be done with expensive, optimized hardware platforms, was now attainable with inexpensive commodity hardware. On top of that, the open source community is rich with effective cyber defense tools at a great price (like free, in many cases). Most of our time was spent building the software framework that could host and manage multiple software-based cyber defense tools (or modules as we call them). Our team of committed engineers, rock star logistics folks, and Government leadership team, brought the whiteboard scribbles to life, and Modularity 2-5492 is currently being deployed to the network.
We still have much more work to do, but the most important part of the strategy has come to fruition – building a framework that can host multiple cyber defense capabilities, and more importantly, a framework that enables our cyber Warfighters to remotely deploy new tools as the cyber threats evolve. Yes, if we need a new tool, there is no more deploying bare iron to data centers across the globe.
Some of the Open Source
- pf_ring – our good friends at ntop have done some remarkable things with packet processing, enabling super-fast capture of packets crossing the network.
- CERT NetSA Security Suite – our good friends at the Software Engineering Institute (SEI) offer a suite of open source tools from netflow collection to application fingerprinting.
- Suricata – a great open source IDS/IPS with some really impressive performance at high bandwidth sites.
If you want to learn more about the nuts-and-bolts of rolling your own cyber defense system, we’re happy to help. This is certainly not a novel design; however, it does take some gluing together of the parts, and of course, if you’re in the Federal or Department of Defense space, we have navigated certification and accreditation (C&A) process with much love and attention. Yeah, remember that analogy to Mt. Everest? Multiply that by two!