Michael Epley, Chief Architect, Public Sector, Red Hat discusses the challenges of secure information sharing. Why cross domain security is key for enabling faster, more secure development. We discuss the challenges, solutions and the tools.

Episode Table of Contents

• [01:20] How Cross Domain Security Is Being Used
• [06:32] A Link Between Restricted Network and Open Network
• [12:25] Throwing Content Across the Cross Domain Solution
• [16:36] The Friction That the Cross Domain Challenge Presents
• About Our Guest

How Cross Domain Security Is Being Used

Carolyn: Today we have Michael Epley, Chief Architect Public Sector at Red Hat.

Carolyn: Mike has been helping the US defense and national security communities use and adopt open source software. He’s been there over the last two decades with practical experience as a software developer and enterprise architect. Today we're going to discuss secure information sharing and the role of cross domain security in doing that.

Eric: We were just talking about skiing as we were prepping to get started, which was great.

Michael: Ski season is approaching and making plans already.

Carolyn: Lets jump right into cross domain security and how you're seeing it being used throughout government and even in commercials. What's it all about Mike?

Michael: There are people who aren't used to working in these environments. Cross domain and cross domain technology is the ability for us to enable working in restricted or air gapped environments. In order to do so, we often have to transfer information, data, software, and other systems from open networks.

Michael: I think of internet accessible resources onto these restricted networks. To do so, we've over the years and the DOD and intelligence communities been doing this for quite a while. Without a lot of specialized technology to enable that, a lot of these specialized technologies are kind of Harbor based. Typically, we have boxes that have paired diodes and photo resistors.

Michael: They take advantage of the physical properties of these systems so data can flow in one direction.

A Painful Memory

Michael: A matter of fact, we often call these diodes because of that. In the distant past we also used CDs or other types of media you can write to once, walk them across the gap, and then read the CDs.

Eric: What we call Sneakernet, right?

Michael: Absolutely. As a matter of fact Sneakernet is colloquialism and unfortunately a painful one from my own memory. I have Sneakernetted many CDs over my years. In fact, I've had lots of experience doing this kind of cross domain development to include some humorous samples.

Michael: So now as a software engineer at Lockheed and I had to do some cross domain development, but I didn't have access to the restricted network of classification reasons.

Carolyn: I'm told that's still a thing, Sneakernet.

Michael: It is and people have developed lots of workarounds for this. Back when I was at Lockheed, for example, I had my manager have access and I didn't. He would go into the skiff and I literally was yelling instructions at him through the wall. Hoping he would be able to type in what I wanted him to.

Michael: He didn't know any software or software development. Then he would try and yell back when he saw the screen through the wall. We did that for about six months, but you got maybe a handful of lines of code written in six months.

Carolyn: Is that legal?

Eric: Wasn't even Sneakernet. That was kind of, that was like a relay race where you pass the baton.

Michael: Sneakernet would have been a luxury. I could have actually gone into this gift at the time, but I wouldn't be. These restrictions can be quite numerous.

Walking From One Network to Another

Eric: Too bad Garmin and Fitbits aren't allowed in the environment. Because you could actually track how many miles you're walking from one network to another over the course of the day. There’s tech that helps make that go away, which helps in development environments.

Carolyn: Eric, you just said something that I didn't realize these networks are physically, really far apart?

Eric: No, but if you're walking from one to another you're walking to the scan station. You'd be amazed how many steps you get when you go from your desk to the scan station. Which is probably not near your desk. It could be hundreds of feet away or more different floors even. You scan it, which is typically an AV scan, at least in the old days.

Eric: There isn't much more, I don't want to get into too much detail. Then you've got to take it into the skiff or wherever you wanted to take it. Which honestly, it could just be going from one network to another. It could be two different laptops on your desktop, or two workstations. But you gotta walk to the scan station with your CD. So you add up the miles, you add up the steps.

Michael: We can thank the GSA for planning out the parking lot. So they're exceedingly large some of these. You do log some miles.

Eric: Anyway, there's a better way to do it, Carolyn.

Carolyn: I'm curious as to how or why Sneakernet is still a thing if cross domain solutions solve that.

Michael: It's still a thing because even with some of our newer, more advanced technologies, they can still be cumbersome to use.

A Link Between Restricted Network and Open Network

Michael: These are typically like I mentioned earlier, a kind of specialized hardware device. They're taking advantage of these physical properties like diodes or the resistors. There's few of these typical enterprises. They're kind of high priority protected enterprise resources that are carefully managed. Not everybody gets to use these resources.

Michael: Typically, they use things that are permanently added to your network. Two things right there. They're enterprise resources, typically reserved capacity for certain use cases. You're streaming data into these ad networks for analysis or other purposes.

Michael: Have other workloads or other users on these systems that are neck to neck manage that priority. Make sure that’s not stopping the machine data from crossing over that network. If you're talking on an aircraft or a restricted network, you're adding a link between that and the open network that represents a security risk and security vulnerability.

Michael: They have to be carefully managed and protected to make sure that connection point doesn't turn out to be a disadvantage or security weakness.

Eric: The worst thing is to let an adversary through what's considered a secure network, which is no longer secure. That's a gap they can cross. That'd be a bad thing.

Michael: That's why security in these situations is paramount. Eric, you mentioned in the olden days there'd be an AED scan or something like that. We have to be careful. An AAV scan is just one step in that process crossing that boundary.

Implementing a Stronger Perimeter Security

Eric: Correct and you mentioned when we were talking before the show started. You're seeing a lot more need in the commercial space. Especially post COVID with work from home and everything. You're seeing corporations, commercial corporations actually set up air gapped in separate networks.

Michael: You're seeing even across the commercial space. There's several industries where this is particularly true. They're learning lessons from the DOD and IC about how to protect their networks, implement stronger perimeter security which is essentially what restricted air gapped networks represent more defense in depth.

Michael: They're taking those concepts to increase their security and boundaries because they've got standards like PCI or HIPAA. Things like these that require higher levels of protection than they once were. They were looking for solutions. They've had a way to do that and the DOD, and the intelligence community. They have been implementing these technologies for a long time.

Michael: Now they have other requirements and of course, huge workforces and all that. They have access to these restricted networks, but even in the DOD and IC either, they're trying to, in the name of security.

Michael: Move their workforces into these little domains to only bring in a need to know or other goals to limit access to those restricted networks. Either by people or technologies, that they can again improve their restrictions footprint.

Carolyn: Who are you seeing in the commercial world? I think you mentioned financial.

Michael: The financial services have been leading that. Mostly because the financial services industry got a lot of resources and assets to protect and they're huge targets.

The Challenges of Cross Domain Security

Michael: There's lots of fruits and it's a highly regulated industry. So the meat, a lot of it's security departments they're either industry bodies or laws or regulations require. They've had to increase their verdict.

Eric: Critical infrastructure, Caroline's another area. Energy, oil, and gas, critical manufacturing, good examples we're seeing.

Carolyn: Are they hitting challenges or the roadblocks that's making it prohibitive for them?

Michael: The challenges and roadblocks you're saying are the same kind of challenges that you see in any kind of a DOD and IC environment. We've been doing this for a long time. The challenges for air gap security and cross domain security is that there's a lot of friction in the process.

Michael: They're not thought of having these highly specialized cross domain guards and solutions. All these extra kinds of controls that you have in that transfer process make it difficult to manage. There's a lot of governance control and roles that you have to follow.

Michael: Whether it's limiting the number of people who have access to the systems or just the actual delay. Or extra level of effort that's required to use those systems. Eric talked about literally walking down to the scan station. That might add 30 minutes round trip plus another 30 minutes to scan something.

Michael: These days especially where we're trying to do agile development, rapid cycles and things of that nature. Even those types of delays can often be too burdensome. Plus to actually implement these specialized crossbeam tools, there's a lot of extra infrastructure to manage.

Throwing Content Across the Cross Domain Solution

Michael: You have to do this across multiple domains. Let's say it's a weak point in your security perimeter. You have to be very careful about how you do that. You're going to have limited people, limited resources to do that with.

Eric: The same thing with the scan station. I remember burning two CDs for bringing a CD in from the outside, walking down, you get basically a virus scan. We won't go into more details. But you're not getting the same level of inspection you can get from across the main system.

Eric: You then walk it to the desktop, you upload it to the higher side network. Then you can let someone know or do something with it, but there's still risk there. In fact, I would argue, there's more risk, but I can't imagine doing it all the time as a developer.

Eric: Every time you have to make a code Mike. I certainly can't imagine yelling through the wall into this gift to somebody who's not a developer, who's not with you. That's a phenomenal story.

Michael: That's exactly what we're trying to do with modern cross domain solutions though. We're essentially trying. You've got a software developer in an open network number. He's the analogy, as he's yelling across hallways, throwing software, throwing content across that cross domain solution.

Michael: Just kind of crossing his fingers and hoping that it works or that it does what he hopes it does. That's where the solution is, to figure out ways that we can provide higher quality assurances. That what he wants in fact gets across that boundary and then that work as intended.

A Deeper Level of Inspection

Carolyn: You mentioned, burning CDs, Eric. Does cross domain solutions prevent things like Lady Gaga CDs from being burned?

Michael: The answer is yes absolutely. Even at these scan stations they will examine the media and look for potential weak points in that media. So multi-track CDs, for example, you can hide data and a second track. You can hide data in the index that's burned off the CD, things of that nature. Specialized tools are needed to scan for those items and check potential issues like that.

Carolyn: That's what cross domain would do? It would give it that deeper level of inspection and prevent something like that from happening?

Michael: Modern cross domain technologies. Especially ones that are implemented with specialized hardware and software systems, as opposed to like a CD. They can definitely do those deeper level inspections. Even more importantly, they can do those things out of down, out of band from what the user sees. Their user might just see a simple file transfer.

Michael: But that file might be held by the cross domain system during that scan process. Then examined by dozens or hundreds of different tools. Or even saved or snapshot and collect to a security analyst, then take a deeper inspection or look at that.

Michael: Even to the point where they might take that software, say they detect potential malware. They might actually run a specific malware analysis against that particular transfer to make sure it's safe. They can either alert it or hunt down the source of that malware after the fact.

Carolyn: How do we get broader adoption of cross domain?

An Automated, Seamless, Rapid Process

Carolyn: You mentioned the challenges and the reasons that it's not being adopted. How do we make it so it can be more broadly adopted?

Michael: One of the big things that's caused some of these systems to be limited at adoption is that they’re specialized enterprise resources. The more we can take these enterprise and specialized resources and implement them using commodity systems technology is hard and even more importantly, software only solutions.

Michael: We don't need to invest in specialized hardware. We can actually make the deployment and use those much broader. There's a number of technologies out there today which either have nice properties for doing so or are actively being used.

Eric: Also the repay off are areas like tactical Carolyn, where you're doing the same thing over and over again. Like position navigation, timing, data, where you want it to be an automated, seamless, rapid process. It makes sense to put it in there. You accredit the system and you move on. Those are some great examples.

Eric: I'm not a developer, Carolyn. I don't think you're a code writer. You might've been from your past. I've never been a developer. I'd love to know what that experience is. Developing low and moving high, especially with work from home.

Eric: How do the people you interface with day to day at Red Hat, your customers, they're working from home. I'm assuming they're writing some level of code at home periodically. How do they deal with that? Then get it back to the IC or the DOD organizations they're supporting.

The Friction That the Cross Domain Challenge Presents

Michael: Yes interesting you asked that. Over my ten years as a software developer, I actually worked mostly on these protected high side classified networks. The difficulty and the friction that the cross domain challenge presents is actually pretty hard to surmount. That’s a huge challenge and not a lot of people solve it.

Eric: Just develop on the high side and then it's easy. You don't have to cross any boundaries. You're just in the environment you're developing and you're going to work in.

Michael: Honestly, those that are in that cross them out domain development, for years, were so high. That system integrator, software developers, integrators, just accept the fact that it was so hard. They would hire and train and get cleared and get resources into these environments and then ask where their primary work location would be.

Michael: Even though it was more expensive, it was hard to find people that slowed down development. You don't have access to the internet, you can't just Google something to find the answer. They would just accept that because it was still easier than doing a crosswind.

Carolyn: How does cloud play in cross domain, or does it?

Michael: Cloud definitely plays in cross domain. Like a software only solution being one big enabler that you had a software only cross-domain solution. You had cloud environments with cloud resources. You could take advantage of that cross domain solution in software, on those gliders, not have to deploy hardware.

Michael: Right now, if you had to implement a hardware based cross main system, you can't do it in the cloud.

What Does the Future Look Like

Michael: Unless you own that cloud yourself, you can't just do it. Amazon is probably changing this. But it’s where you can have private clouds of your data centers. You typically can't put a hardware in somebody else's data center.

Eric: You have to be the cloud service provider, basically. Which is good. There are lots of good options out there for developers. So what does the future look like? Like go down the road a little ways. What would you like it to look like? And what do you think it looks like from a developers perspective?

Michael: I mentioned those boundaries are really high and it does have that all those frictions cause a lot of challenges that make things more expensive and slower. We're seeing software development move into a world where it's rapid deployments at CIC. It's very agile and we want faster.

Michael: As a software development developer, that's what I want. I want to be able to make a change, compile it, run it, test it, deploy it, see how it works.

Eric: Change it again, iterate on that?

Michael: Exactly. The reason why we need cross domain solutions for this is if you're testing on these restrictive networks against restricted data that you don't have access to. You have to be able to transfer that software. And you have to be able to do those builds and tests and appointments on those restricted networks. Even though you don't have that in a bar.

Eric: Have you seen it get better over time? Have things become more fluid?

Michael: They're definitely getting better.

Better Career Advancement Opportunities

Michael: Before we started this conversation here, you mentioned several prime examples of programs and people doing that. It is getting better. It’s being driven by some of the recent changes in the kind of workforce and environment. That competitive pressure I've mentioned, it's expensive and it's slow.

Michael: The people that can solve that problem are going to have a competitor. Especially as the private sector IT and software and systems get better and they pay more. They have better career advancement opportunities for software developers. They’re generally more attractive and fun or places to work.

Michael: So the DOD, the intelligence communities, the people operating these restricted environments. If they want to compete with the private sector for talent and for people, they have to be able to do this better.

Eric: We were talking about that in COVID time. A lot of people are working remotely these days. We were talking before that, before the show. In the examples you gave Mike, a developer could literally go to the beach. They could go skiing.

Eric: They could go somewhere. Continue working during the day in a new spot and then submit their code. Move it through, move it up to the high side, have people work on it, test or come back for that piece. But they can work from wherever. We can take advantage of a lot more of a workforce.

Michael: You know, that’s COVID. The work from home and quarantine kind of environment is really pushing people to be faster. To kind of figure out what technology to purchase because they are forced to. They'd had a workforce that was used to working in these restrictive environments for a long time.

Getting a Developer’s Perspective

Michael: They literally have to solve this problem overnight, or they will fail to get their mission done. So they are working very hard to figure out after that and I mentioned it in the USOC. There are modern technologies that we can turn to help solve balances.

Eric: I was reading an article. It was in the New York times, over the weekend. Facebook, Twitter, a lot of the Silicon Valley companies, their developers, their people are moving. They want to get out of the Bay area because of COVID because of the cost of living and everything else. The author was pointing out one of the inherent benefits.

Eric: It’s that there's an entire component of the country that they haven't been tapping from a workforce perspective. I almost feel like with some of these initiatives on the DOD and the IC side, the gray box with NSA and in some of the others, there's a whole workforce we're not tapping that might be interested in serving the country if they could do it more easily.

Carolyn: It sounds like we're getting there. You know how I love to end on a high note. So thank you very much, Mike, for joining us today.

Michael: My pleasure. It was great.

Eric: It was great getting the developer's perspective.

Carolyn: You're one of the first developers we've had on.

Michael: Well I like to think I'm a developer but I honestly don't touch much these days.

About Our Guest

Michael Epley, Chief Architect, Public Sector, Red Hat. Mike has been helping the US Defense and National Security communities use and adopt open source software over the last two decades. With practical experience as a software developer and enterprise architect.

During his tenure at Red Hat, Michael has passionately driven adoption of key technology. Cloud and kubernetes, tactical edge/forward deployed systems, data analytics tools and platforms, and disconnected operations.

Always in the context of security and compliance concerns unique to this sector. Michael has BS degrees in Mathematics and Mechanical Engineering from Virginia Tech and a JD from The University of Texas School of Law.