The State Of The Software Supply Chain, And What It Tells Us About The Adversary - Ep. 104

Derek Weeks is a huge advocate of applying proven supply chain management principles into DevOps practices to improve efficiencies, reduce security risks, and sustain long-lasting competitive advantages. He reviews the State of Software Supply Chain report

Please give us a review, let me know you gave us a review via LinkedIn https://www.linkedin.com/in/carolyn-ford-3b891a3/ and I will send you a free copy of "The Talent War"

Episode Table of Contents

• [02:21] What Is Software Supply Chain
• [08:41] The Threat of Software Supply Chain Attack
• [16:37] The Difference Between Automation and Scavenger Hunt
• [18:15] Understanding Our Software Supply Chain Better
• [21:07] Develop Software Quickly and Keep It Secure
• About Our Guest

Give Us a Review and Get a Free Copy of “The Talent War”

Eric: We have Derek back. I'm so excited.

Carolyn: I know. And I want to say hello to our listeners really fast. Because I want to do my plug of giving us a review on your podcast platform. And this week, if you give us a review and then share that review with me on LinkedIn or tag me somehow, you can even email me cford@forcepointgov.com, I will send you a free Talent War book. So that is George Randle and Mike Sarraille's

Eric: From Echelon Overwatch.

Carolyn: So, their book's coming out November 10th. You give us a review, I personally will send you a free book. So there we go. And now let's get to our guest this week. Derek, thank you for coming back.

Derek: Carolyn, Eric, thank you for having me back. Our last conversation was pretty fun. So it's always enjoyable to be here with you guys again.

Eric: I learned so much.

Carolyn: Same. And so we talked about DevOps and DevSecOps last time, and you did a mean thing to us and left us hanging at the end of that episode.

Derek: That's what all episodes should do.

What Is Software Supply Chain

Carolyn: It was great, which is why we needed to have you back.

Derek: That's the whole thing. Right?

Carolyn: Exactly. So, the teaser was the top attack you've seen through the software supply chain based on the latest State of the Software Supply Chain Report. So, I want to jump in, but first, will you set the stage for our listeners? I honestly had not heard of the software supply chain before. I've heard the supply chain, not the software supply chain. So can we just start there and just give us a quick overview of what that is?

Derek: Yes, I think that's a great place to start, Carolyn. So, I think the easiest way to look at it is everyone has technology supply chains. And within those supply chains, you have hardware and you have software. But those supply chains operate in very different ways. So, if you think about a normal supply chain, technology supply chain, there are suppliers out there that make computer servers, network equipment, storage, et cetera.

Derek: And if your organization wants to purchase those, they go out, they talk to the different suppliers, they figure out, "This is what we need. This will be the price of it." And normally you work through a procurement organization that will say, "This is a trusted supplier for us. We vetted them. We have them in our financial accounting systems, et cetera. And we can process this and we know exactly what we bought, who bought it, and where it's going in the enterprise."

The Modern Software Development

Derek: In terms of software supply chains, if you buy an enterprise package software, it kind of operates in the same way. But there are other parts of a software supply chain that are less transparent to you if you're not a developer. So, in modern software development, there is a huge, vast ecosystem of suppliers out there for software. If you're a developer, you don't code your applications from scratch anymore. Effectively, what you do is assemble most of your applications from open source components. And those components are built by open source projects. And that is a single developer or a team of developers around the world that contribute code and make it freely available.

Derek: These are the suppliers of software in software supply chains. Containers work the same way, that anyone, you or I, could build a container, put it up on Docker Hub. You or I could build an open-source component and put it up on Maven Central or NuGet Gallery. And any developer in the world can download that software component and bring it into the organization. It does not pass through procurement. It does not get vetted by anyone in the organization unless you have automation set up to do this.

A Huge Volume of Consumption Software Supply Chains

Derek: But just to give you a sense, in an organization developing in Java today, that organization will download about 300,000 Java components through its software supply chains that don't normally get vetted by anyone in a normal procurement kind of process. If you have 1,000 developers, you have 1,000 people in procurement for your software supply chains. If you are developing in JavaScript. My research this year in the State of the Software Supply Chain Reports to every JavaScript developer on average is pulling in 100,000 open-source software packages a year. So there's just a huge volume of consumption software supply chains.

Eric: I want to ask a question. Every developer pulls in how much?

Derek: Every JavaScript developer on average pulls in 100,000 JavaScript packages per year.

Eric: So, in the beginning, in the introduction to the report, you said that corporate software engineering teams have 20 million software developers.

Derek: There are about 20 million software developers around the world.

Eric: So when you multiply those numbers together, I mean that's a massive number.

Derek: Yes. So, just in the JavaScript realm alone, there are about 7 million JavaScript developers in the world. They will download about 1.2 trillion open-source packages this year. And those do not go through the normal procurement technology supply chain. They just pick the number of developers you have and that's the number of people effectively in procurement for your software supply chains.

Eric: Wow. So they're just pulling them in and nobody's doing security checks necessarily.

How Many Developers Follow the Policy

Derek: Some organizations are. The high performing organizations are. But we go out and we survey different software development organizations throughout the year. And this year we looked at, or we surveyed over 5,000 developers, and about 57% of those said, "We have a policy in place for what open source we can pull in or not." But when we asked them, "Do you have a policy, and do you follow that policy?" A smaller percentage of the developers actually follow the policy, because often if it takes weeks to get approval for an open-source package that a developer wants to use, they'll usually ignore the process and work around it.

Derek: So, the actual compliance to the policy is well under 50% of organizations out there that have developers that are following rules for what they can bring in to organizations or not. And some have absolutely no rules. Developers can consume any components that they want.

Eric: Wow. That sounds like a lot of risks, Derek.

Derek: It can be, certainly. And I think we've seen that playoff or play out in a number of different scenarios, whether it be the Equifax breach, OpenSSL, and the Heartbleed vulnerabilities that were out there along with other large open source vulnerabilities that have been introduced in the last several years.

Eric: I was looking, I have it right in front of me here. I know Carolyn's having some mic issues, but I was looking at the report, reviewing it again this weekend. I read it before, but the next generation software supply chain attacks, 2015 to 2020, the magnitude, I'm going to hold it up.

The Threat of Software Supply Chain Attack

Eric: I know we're doing some video these days. The magnitude of the increase is significant, really starting in, I don't know, a year, a year and a half ago, June of 19 and then another one in December of 19. Just boom. I mean, if that's your batting record in the world series, you're doing well. If this is the number of attacks hitting your business, you should be scared to death.

Derek: Yes. So, I'll throw out some scary information because, I think, for our listeners, it's really important to understand this. If you look at an open-source related breach like Equifax, I mean, this is the poster child for it, it was an open-source component they had, it became vulnerable. The adversaries found it in a matter of three days and the initial breach there started. And it also started in about seven other places around that same day that Equifax was breached.

Derek: But this is kind of a legacy style software supply chain attack. Adversaries are waiting for an open-source component to become known vulnerable. Once it's vulnerable, they race to find instances of that vulnerable component that are deployed out there faster than the enterprises can patch it or update it. So, since Equifax, there's been a considerable investment in organizations saying, "I don't want to be the next Equifax. And how do I avoid that?" And they're investing in things like software composition analysis, that is aimed at using the best, most secure open source components out there.

Good Code and Bad Code

Derek: And in the annual survey that I mentioned, we've actually seen the percentage of breaches attributed to open source drop from 31% of our developers shortly after the Equifax breach to 21% of developers that we survey. So things are improving there.

Eric: Because of awareness and action?

Derek: Organizations are investing more in, "I don't want to be the next Equifax." So the stats that you pointed out on the report of the increased number of attacks actually show a change in adversarial behavior. What used to be a kind of wait and pray technique, "Let's wait for the vulnerabilities to come out and then exploit them." Adversaries are saying, "Hey, I can be part of an open-source project. I can contribute code into that project." That's then supplied to developers around the world.

Derek: So if I pick an open-source project as an adversary to contribute to, where I can contribute good code and bad code, that open-source project may be downloaded 10,000 or 100,000 times a week. If I insert some malicious code into that, it can be deployed or downloaded 100,000 times by developers this week and then put into applications where I'm effectively constructing my own zero-days, organizations aren't paying attention to the quality of open-source that they're pulling in. And this is where we saw an increase since last year, we saw a 400% increase in this type of attack. There were 900 attacks since June of last year, since June of 2019.

How to Secure the Software Supply Chain

Eric: So, that was my question. I was actually listening to the recording you made last night to refresh myself on the way back from a family trip. And that was the real question. As an adversary, it seems like you would have a huge incentive to contribute good code with malicious code buried inside of it that only you know about, which you can then monetize or exploit. So I'm a government software development manager. What do I do to figure that out to prevent that? Because obviously, I need to use open-source software. So, how do we secure that software supply chain, as you refer to it, to make it better?

Derek: Well, I think that the first and most basic thing that you need to do is get an understanding of what open-source you're currently consuming within your agency.

Eric: And I'm betting most people have no idea.

Derek: It all starts with, "We need to talk to our developers and we need to work on ways to assess or get an inventory of what they're using." And getting an inventory is difficult if you approach this manually. We just talked about every single JavaScript developer downloading 100,000 packages. You could never employ enough people and manually automate that. Now you can actually just do a review, a software built in about five seconds or 10 seconds to say, "Give me a report," or create a software bill of materials to say, "What are all the open-source components that we've used in this software?" And creating that inventory gives you then the next to understand, are those parts good or are they bad? If you don't know what you're using, you can't evaluate whether they're good or bad.

Eric: Awareness.

Focusing on a Software Bill of Materials

Derek: So you just have to start with a very basic step of identifying what you have. And this is something that it's not my idea per se. I've been pushing this for a long time. If you listen to me out in various public forums where I am presenting and participating. But the Department of Defense is very focused on this. Congress has introduced legislation on this. FDA has policies that focus on creating a software bill of materials. There's a large initiative led by Allan Friedman at NTIA as part of the Department of Commerce.

Derek: So, this is not a new topic. This has been around for a while. But there's certainly encouragement across different government agencies to advise organizations to start producing a software bill of materials. And then, as part of that practice, it's effectively creating the inventory, but evaluating if the inventory is good or inventory is bad.

Eric: Wow. So, I just, I put myself in the position of a software development manager and it almost feels hopeless at this point. Derek, we lost Carolyn's audio. She's trying to recover it. But let's continue. Do you mind if we play speed round?

Derek: Yes.

Open-Source Software and Its Vulnerabilities

Eric: I'll throw some statistics out from the report and you give me your quick thoughts.

Derek: Yes.

Eric: So, okay. So high performers detect and remediate open-source software vulnerabilities 26 times faster, according to your report, what does that mean?

Derek: So, what it means is they're actually paying attention to what open-source they're using. They've applied automation in a way that tells them, "Hey, if there's a new vulnerability that's announced today, we know what we've used. If we use that component. And we know where it is." And automated intelligence tells us if there's a safe version to upgrade to and allows us to do that rapidly. So that's how high performers end up 26 times faster than the low performers that we've studied.

Eric: You also said 51% are more likely to create a software bill of materials. I'm assuming that the software bill of materials is that catalog they use to know what they have and where it is so that when an exploit is announced, they can address it quickly.

The Difference Between Automation and Scavenger Hunt

Derek: Exactly. So if you think back to Heartbleed or the Struts vulnerability that was at the center of the Equifax breach. The first question that you have in hearing about this new vulnerability in an open-source component is, "Did I ever use that?" The second question that you have is, "Where did I use it, and if I did?" And you either end up with automated technology telling you, "You used it, and here's where it is. And it's vulnerable, and there's a safe version." Or you end up in a scavenger hunt. "I have no idea if we used it, let's go create some search parties and go out and look manually at all the applications in our portfolio." So it's really a difference between automation and a scavenger hunt.

Eric: So, that sounds like a nightmare. Okay. So here's another one that blew me away. And I think back to some of the hardware exploits I know, I think the most famous whether accurate or not is the Super Micro report that I think Bloomberg released a year, a year and a half ago, on the actual bias being compromised on the Super Micro board. But 11% of components used in applications are known vulnerable, coupled with nearly 40% of all npm packages, excuse me, I almost said RPM, rely on code with known vulnerabilities. We know there are vulnerabilities in our open-source software, which is what you're telling me, at a significant level.

Understanding Our Software Supply Chain Better

Derek: Yes. I was in a panel discussion last week with Katie Arrington from the Pentagon, Department of Defense.

Eric: Yes, CMMC.

Eric: She's been on the podcast. She's great.

Derek: Yes. So, she brought up this idea of, "We need to better understand our software supply chains." And I had brought up during that conversation, hardware attacks, whether the big hack in Super Micro actually happened or not, it's extremely hard to pull off. You got to get it to someone in the factory to get an electronic component embedded in a circuit board, have that circuit board go out, not be vetted through the technology supply chain. It's hard to infiltrate that factory.

Derek: By comparison, software supply chains are not difficult to infiltrate. They are created by a bunch of people like us, that get on an open-source project, contribute code and that's how all of this gets created. So it's pretty easy to become a member of the community. It's very easy to contribute to the community. And if you're someone with malicious intent that wants to contribute bad code, it's pretty easy to go about doing that.

Inject the Vulnerabilities

Eric: So, I'm reminded of the articles you've seen, the research I've done on the Chinese has been very prolific. They've joined a lot of standards bodies to drive the standards in their direction. It's a great strategy. Why not do the same thing with open source software? You can inject the vulnerabilities. It's a lot less costly than hardware. It sounds almost easy. As an adversary, it sounds like a great idea. And it sounds like what they're doing.

Derek: Yes. I'm sure that nation-states and rogue organizations are out there doing this today, just on the volume of what's happening out there. I can't say that Russia or China or North Korea are doing this, but really anyone that is a software developer could contribute in this way. And there's a lot of money to be made in cybercrime, and using these malicious code injections into open source to steal credentials, steal passwords, install cryptocurrency miners, et cetera. So, if you're an adversary, you take the path of least resistance.

Derek: Where are people looking the least for these vulnerabilities that I can quickly exploit and make money off of that or get to a target that might otherwise be more difficult to get?

Eric: Yes. Why not? It's easy. So, Carolyn feels like she's in a presidential debate. She doesn't have her mask on right now because she's isolated at home, but she's hearing us and can see us, but her audio is out. But she wants to know, last question, same thing I was going to ask and wrap up. I want to know what orgs should do, organizations should do to keep their supply chain safe. What are the number one things? You already said, understand what you're using and where.

Develop Software Quickly and Keep It Secure

Derek: Yes. So, that's part of it. Secondly, is you really need to look at automation in these areas. So there's a set of tools called software composition analysis, the Gartners and Forresters and others of the world cover this space. And I would certainly look into that. There are a number of free tools available on the market that can help you evaluate applications. If you're leading software and development initiatives, there are of course paid enterprise tools that are available out there.

Derek: But the other thing that I'll just advise is that a lot of organizations may say, "Hey, we're moving really fast with software development practices and DevOps, and adding a security is going to slow us down."

Eric: Of course.

Derek: Or there are organizations that say, "I've got security baked into my development life cycle. If I move any faster, I'm going to be less secure." And part of what we cover in the State of the Software Supply Chain Report is new quantitative evidence, academic rigorous research that says you can actually develop software quickly and keep it secure. And that the highest performers out there are more secure and faster than those emphasizing security first as a practice or emphasizing productivity first as a practice, there's actually a high-performer group that can do both. It's about 25% of organizations we surveyed this year that are doing that.

Join the All Day DevOps for free!

Download the Report

Carolyn: Derek, your 24-hour conference All Day DevOps happens in two days, November 12th. Are you guys ready?

Derek: We are ready. We've got about 50 people working behind the scenes at this conference from around the community. It's free, it's online. We'll have about 30,000 people attending this year. There's a government track. There is a DevSecOps track. And four other tracks that we have in the conference. So definitely join in, participate. It's free online and invite everyone in your organization to participate as well.

Carolyn: Listeners, you can download the State of the Software Supply Chain and see the tips that Derek was talking about, about how to keep your software supply chain safe. This episode was a little bit crazy. I had technical difficulties a few minutes in, I lost audio. And I could see Eric and Derek speaking, but I could not hear them. I was able to just listen to the episode though, and just a lot of good information. So download the report.

Carolyn: And also remember if you give us a review on your podcast platform and then let me know via LinkedIn, I'm Carolyn Ford on LinkedIn. I will send you a free copy of The Talent War. And we just reviewed that book with author George Randle a few episodes ago. It's such an excellent book about hiring for success. So give us a review, let me know, and I will send you a free book. Thanks for joining To The Point Cybersecurity.

To The Point Cybersecurity was recently named one of the 30 top Federal IT influencers  2019 & 2020  because of fantastic guests. We are always looking for great thought leaders to interview. Please email me with guests you would like to have on the podcast cford@forcepointgov.com

About Our Guest

Derek Weeks is a huge advocate of applying proven supply chain management principles into DevOps practices to improve efficiencies, reduce security risks, and sustain long-lasting competitive advantages. He currently serves as vice president and DevOps advocate at Sonatype. Derek is the co-founder of All Day DevOps, an amazing virtual conference bring together DevOps practitioners and thought leaders. It’s the largest virtual conference in the world, educating DevOps professionals through online training and blog content, and host over 180 local community events in 20 countries around the world. Since its founding in September 2016, our community has grown to over 130,000 strong.

This is the first year we will have a government track- 18 sessions (all thought leadership) State of the Software Supply Chain Report: https://www.sonatype.com/2020ssc