[00:53] Software Supply Chain Advocate
Rachael: Hello everyone, welcome to this week's episode of To The Point Podcast. I'm Rachael Lyon with my new co-host Petko Stoyanov.
Petko: Thanks for adding me as a co-host. This is an amazing podcast with amazing guests and I'm looking forward to today's guest.
Rachael: Welcome back to the podcast Derek Weeks. He's a software supply chain advocate. He is one of the world's foremost researchers on the topic of DevSecOps and securing software supply chains. Such an exciting conversation today, Derek, I can't wait to get into it.
Derek: I wish we had solved all the world problems that we had discussed in our inaugural episode together. But, there are still some things that are left to be done in this area. Petko, it's nice to meet you as well.
Petko: Pleasure to meet you, Derek. I think the last time we had you on this podcast was actually before COVID. BC and AC are how I see life now. So, we haven't solved cybersecurity since COVID.
Derek: No, it's still rearing its ugly head every now and then. So, it's best that we continue to talk about it and help people through their journeys.
Rachael: This is one of my favorite topics. I swirl around this idea of grading company security. You think about things like this Open Source Software Act of 2022 that's coming up. Companies have gone from 35% to 75% of open-source software as part of their audited code base according to McKinsey. That's pretty significant. If you don't have any parameters around this right on the supply chain, then how can you feel confident about what you're bringing into your company? That what you’re building your business on is going to be safe?
The McKinsey Study
Derek: I think there are a lot of people paying much more attention to their software supply chains than they ever have before. I think that’s a good thing, but there's still a huge amount of work that needs to be done in this space. Part of what you mentioned, is the McKinsey study. It says, "Everyone's using open-source software or open-source components in their software."
I remember, going back almost a decade when I started working at Sonatype. At that time, there were let's say 20 million software developers on the planet. They were running, they still run Maven Central which is where all the Java open-source components exist. The year I joined them, they had like 13 billion downloads of Java open-source components from 20 million software developers.
I thought, "There's just a huge amount of activity happening." Even if you go to the extremes of how many software developers there are today. It's somewhere I've seen between 50 and 70 million software developers. So, let's assume that's true. The number of open-source components being downloaded annually is something like 3 trillion. If you do the math on how many open-source components are being downloaded and used, it's no surprise that any modern application has open-source software in it.
Developers are choosing to download the code rather than write it themselves. There are massive innovation efficiencies afforded to organizations that are using these. So, there's a lot of good happening in this. It comes down when we look at software supply chain security. At the heart of the discussion is if you're going to use an open-source component, is it good or bad?
SBOMs as Part of Modern Software Development
Derek: What have we done to help you make that determination? Good or bad could be a cybersecurity vulnerability or it could just be as outdated. It's buggy, it doesn't run as well as the previous versions or it's not as feature-rich as the previous versions. I know you mentioned the Secure Software Supply Chain Act, we can get into that. But I think just setting that context for people that aren't totally aware of open-source usage out there.
It’s just how much it has proliferated modern software development. You have to recognize the scale at first. This isn't our problem that's going to happen or a problem or a behavior that's going to be ubiquitous. It has been ubiquitous for a decade, it's only getting more and more so.
Petko: When you start talking about scale, it's hard to imagine trillions of downloads. But you were involved with the Linux Foundation. Just thinking of the scale that had everything from cloud fan foundry to cloud native computing. It allows you to do multi-cloud security to be involved in blockchain to the software that runs on your TV.
That's all open-source, that's a project that was used for someone to build on top of. Think about it. The application that's on your TV at home today started with an open-source project and is still supported as an open-source project.
Derek: Yes, and that is allowed for massive waves of innovation and every single imaginable market. From genetic sciences to farming and agriculture to the next-generation energy grid.
Most Critical Open-Source Projects
Derek: From infrastructure to cybersecurity, to cloud computing, to the Linux kernel, it is all out there. Some of it is used much more than others. But some of the most critical open-source projects only have a couple of developers that are working on them. It’s not because they're not super popular.
But a couple of developers are working on some critical piece of infrastructure, there are 10 of them. There don't need to be more than 10 people necessarily working on those and it's fine. But not all of those developers might have the expertise on the security side that we might like them to have.
Petko: I can't help but think about Log4J and open SSO and some of the changes that snuck in there. What should organizations or what is the industry doing to ensure that open-source is secure? I'm sure at least we can audit it publicly, which is great. But is there something that we should be doing? Something above and beyond what we're not doing today as an organization, as a nation, and as individuals.
Derek: I think we go back a couple of years, we could just replay the parts from the previous podcast. I said, "Here's what we should be doing that we're still not doing enough of today." So, one part of that, there are trillions of downloads from tens of millions of developers. The thing that we want to be aware of first is what we’re putting in our software in any application being built today.
Creating SBOMs to Assess Vulnerability
Derek: What have we put in it and is that good or bad? One way to assess, the industry has come up with this so it’s not new, is creating a software bill of materials. Which is effectively an ingredient list of all the open-source components and their dependencies that went into building this app. If the SBOMs are also associated with vulnerability information about those open-source components, then the good or bad determination is specifically to cybersecurity. We can help surface that information to organizations.
Now, that is a great effort. It has been advocated by a number of vendors and a number of people in government, and a number of people in the industry. But when we look toward practice, and again, like I was talking SBOMs eight, nine years ago. 47% of organizations today are using SBOMs. If we look at the number of organizations that are using SBOs across all segments of their inner application portfolio, it's 14%. So yes, we know that SBOMs exist, yes, not almost half of the organizations are using them.
Then you look at how many organizations have an open-source policy that is set on top of using an SBOM. Do you even have rules for your developers regarding whether you can use open-source components or you can't? If you can, what's qualified as a good one versus a bad one? That's your open-source policy. 49% of organizations have an open-source policy in place. When I used to run the DevSecOps community survey, I think I did that for six consecutive years.
[11:03] The Number of Organizations Using SBOMs
Derek: The number of organizations with an open-source policy that was using them hovered around that 45% to 50% mark. This says that if you go back to 2015 and you say, "Open-source policies are important. We're going to see if anyone's using or applying one of these from 2015 to 2022."
There hasn't been any kind of dramatic change in that. Yet, you go back to 2014, which if I remember correctly, 2014 was open SSL and a heart bleed. If you go back to 2016-2017, the major struts vulnerabilities that came out and the Equifax breach that was tied to that.
Go back not even a year ago to the Log4J vulnerability. How many of these things will it take for you to say, as an organization, "We need to pay attention. Set some ground rules for what we can use and what we can't use. Make that available to the developers and the operations teams and the cybersecurity teams there.
It's a bit of a take on where we are which is, honestly not great. If you go out and question people, do you care about cybersecurity or do you care about open-source security? I think the overwhelming answer is absolutely yes, this is critical, and we all need to pay attention. Log4J was such a headache for all of us. Then you look at practice tied to those opinions, it's really lagging.
Petko: During Log4J, I remember the old cybersecurity saying, "Do you know what's in your network? Do you know what it's doing?" When you think of SBOMs, there's open-source, do you know what's in your code? Do you know what it's doing and what its vulnerabilities are?
Did Anything Change with Log4J?
Petko: It also applies to code that same outage. But I know at Log4J talking to so many vendors and even customers, they’re struggling to even know what their internal teams were doing. They weren't even aware they were using Log4J. So, do you think that Log4J almost did it change anything?
Derek: Yes and no. It moved the needle for some organizations, more investment is taking place here. Look at software composition analysis. The key part of open-source software development and cybersecurity is to be able to assess the risk associated with these components. It's moved from the earliest adoption stages eight, 10 years ago to beyond the peak of inflated expectations, beyond the trough of disillusionment.
It is now going mainstream. It's evolved over that time and the markets have evolved over that time. The vendors in this space are now billions and billions of dollars because there is a market in this space. People are paying attention to this, but there is not enough activity.
When you talk about the reaction to Log4J, what's on your network or what's in your applications and people struggling? Going back to the SBOMs, the whole reason that you have an SBOM is when Log4J announced that there is a vulnerability. That X number of versions of Log4J are vulnerable. The first question that any organization has is, did we ever use those versions of Log4J?
If the answer is yes, the next question is where did we use it? And if you can't answer that, you are in really bad shape. You're now going on a scavenger hunt which can take weeks to months to complete.
How Fast Can We Move Fixed Code?
Derek: You can answer it if you happen to have implemented SBOMs and you were tracking what you were putting in your applications. Then it becomes a response to we know we've used it, we know where it is. Now, how fast can we move fixed code or updated code into production?
In a lot of DevOps organizations, there are times they have new code that needs to move into production. The best organizations can get that done in a day under normal processes. Maybe leading organizations that are not necessarily the best could get new code out in a matter of three days. The worst organizations are getting new codes out in a month or so. Now, in that same kind of scenario, what are your adversaries doing?
When Log4J was announced, it was within hours that attacks were being waged. There was something like an estimated 10 million attack attempts made an hour from adversaries after Log4J.
It's not just about what we know. Did we ever use this? Where is it? If we did, how fast can we remediate it? Part of that is how fast our development and release efforts are. How fast are your adversaries at the same time? Say your adversaries can find a new exploit or a new vulnerability and exploit that in under 24 hours. But you can't release new code for 72 hours, so you're vulnerable for 48 hours. There are 48 hours of pain where your data and your customers are at risk. That's really the concerning part of what we face today.
Petko: I thought the US government after Log4J passed an executive order out there.
Pushing the Number of Users of SBOMs to 100%
Petko: I think it was 14028 that required the federal government, if they were buying software, to come with an SBOM. Did that change anything? Is it making the industry start thinking about delivering that as part of a built-in requirement?
Derek: Here's the good news about the government acting. One is when the executive order goes out from Biden or when the Secure Software Supply Chain Act is moving its way through Congress. Many more organizations are paying attention to this. Are we going to be mandated by law to take action? The 47% who are using SBOMs today, optimally should notch up to 100% or have implemented some type of SBOM technology.
But from 2015 to 2022, the needle didn't move significantly even though we had these major vulnerabilities that led to very large breaches out there. When more people pay attention, more action is taken. It does move the needle, so that's actually a good thing.
We looked at the pros being people are paying attention. The con side is, even though I have an SBOM, how fast am I compared to my adversaries? If I've rolled out new software in an application or an Edge IoT device, is it upgradeable at all? 60% of the vulnerable versions of Log4J are still out there.
The infrastructure that they're sitting on can't be upgraded. So, then you have to put in other security mitigations if you can at all.
Petko: You're talking about technical depth if you think about if an organization's not keeping up with the vulnerabilities.
Running a 10-Year-Old Code
Petko: If they're running 10-year-old code, it's going to get harder to get it up to the latest. They don't have to just change Log4J. They're going to change probably 30%, 40% of the code just to get it up there. Who should be responsible for this? I hate to use the word hygiene or the technical depth here, is it the organization? Is it the developers? If we live in a DevOps world or DevSecOps, who owns second ops?
Derek: Hygiene is a really good way to think about this. I use that term often so I don't think there's anything wrong with it. In terms of who's responsible, I think it is the organization overall. It is not necessarily the software developer, but the individual software developer. I think there needs to be a culture of security. You need to combine that culture or instill that culture of cybersecurity. Secure coding, and using the best and latest open-source parts if you will. Combining that with automation in the environment to help developers. I like this.
We talk about software supply chain security, let's look at supply chain security and go back to the manufacturing realm.
Let's go to Toyota's production line. There's some work that's saying, "I need to put brakes on this car going by on the production line. So I'm going to pick the brakes that I've been using for a long time. I'm going to put them in this car. The next car’s coming through." Is it the person on the production line's responsibility to say, "I need to put the best quality breaks on this vehicle going through the production line?"
No, there's someone in the organization that's assessed.
[21:59] The Suppliers With the Best Track Records
Derek: What are the best breaks from the best suppliers with the best track records out there that we can source into our organization? Say, these are the ones that everyone should use. Effective procurement is that kind of policy mechanism for governance policy mechanism for an organization like Toyota. If you take that into software supply chains, today's developers can pick almost any open-source component they want.
They can download it from the internet for free, and put it into their application.
If you're one of the lucky organizations that have an open-source policy that people are actually following, maybe there are some criteria that you can evaluate against. But the number of components each developer is using annually is so large. You have to rely on automation that gives you information to say, "Developer, this is a good component.
By the way, it's the latest or almost the latest version of this component that was released. Therefore, it has the most functionality and the least bugs. By the way, it also has no known security vulnerabilities associated with it, another plus, let's go and use that."
Or, "Developer, this particular component is like six years old. There are 58 newer versions of this component available. By the way, all of the 12 surrounding versions of this component, this one has known security vulnerabilities." Would you move 18, or 20 versions up? Use this better, safer, higher-performing component and give them that information in a simple way while they're coding.
How to Instill a Culture of Security With SBOMs
Derek: Almost like if I'm writing a blog, Google is going to tell me, "Derek, you spelled that word wrong again." Then it gives me a little fix with just the right mouse click. Find the good spelling of that word and fix it. We need to make it so easy for developers who don't have time to go out and manually assess every component they would use or download.
Automation has to be part of that. But again, we have to instill a culture of security in these organizations. Doing that kind of performance check on something as simple as if this thing is good or bad? It’s also as simple as the other kinds of security hygiene. Like, don't use password 123 on anything that requires someone to log in. Don't include code secrets in the releases of these applications.
Eery simple cybersecurity hygiene practices need to be extended beyond the basics that have led to a number of breaches.
Petko: It sounds like we need to put a little bit of guardrails around some of the developers. Otherwise, they'll go pick the latest or greatest version they're comfortable with without considering security. Is more training what we have to do with them or is it regulation? I'm thinking at the national level or the global level.
How do we move the needle as you put it since we haven't in the last seven years? Any suggestions?
Derek: There's no silver bullet to software supply chain security, it is a multilayered approach. There is culture, there is research and education, and there are secure coding classes. There’s automation that surfaces this information.
Implementation and Enforcement of SBOMs
Derek: This is the implementation and enforcement of the policy, there is government regulation that comes into all of this. There is researching and finding the most critical and secure or the most critical open-source components out there. So that we can get more eyeballs on them and secure those components that are being used. I read something like there were 60 or 70 million new projects added to GitHub last year.
Honestly, should we look at all of those for any kind of security flaws? Sure, in reality, you can't do it without automation for one. But let's just say not all of those are supercritical. How do we find out of those 60, 70 million, maybe there are 10,000 or 20,000 that are incredibly important that need more eyeballs? They need a higher elevated level of cybersecurity hygiene applied to them.
What can we do to invest there? Part of it is when we get to the investment areas at the Linux Foundation. The open-source security foundation that's a sub-foundation of that back in May. It introduced a 10-point mobilization plan for securing software supply chains.
The cost outlined in that mobilization plan was about $150 million over two years. The industry should be able to invest in the most critical things. Things that would make a very large impact on improving the security of code flowing through our software supply chains. To some, it could feel like a pretty big price tag. Look at the cost of a Log4J happening. I haven't seen any specific reports on how much Log4J cost the industry.
The Huge Cost
Derek: But knowing all of my friends that were like, this thing broke on. I forget if it was Wednesday or Thursday of the week. It was Saturday and they were texting me like, "My team's been here 24/7. It's Saturday. We're still working on trying to figure out what's been impacted, where we might have breaches." Multiply that by the 100,000 organizations that we're dealing with that, the cost was huge.
Now, this is the open-source security foundation working with industry in the private sector. It’s also bringing in government to say, "We also need your help and support and direction on implementing this plan." But the time has come for us to collectively act, especially in the private sector on really fixing this problem or making a major dent in it.
In reality, when you look at the alternative end of the spectrum, the government comes in with the Secure Software Supply Chain Act. It says, "Industry should implement SBOMs. Here's what we want you to do about it." Whether they covered all the bases or not and we can get to some of the things they didn't cover in that.
Government comes in when the industry fails to act. This is not a good thing for the private sector, this is a forcing function for the private sector. But it's only coming in because the industry wasn't acting fast enough. The public and our data are at risk.
It's not really if you're a cybersecurity vendor offering solutions in this area. The government forces everyone's hand to say, "Now you have to implement these practices." You're sitting very happy because now everyone has to buy your solutions.
Supporting a Common Good
Derek: The reason that that's happening is, the industry had chance after chance to open SSL struts and bash bug. Log4J acted and failed. The government's now saying, "Because you are failing to act, we are taking action." It may be positioned differently like, "We're all coming together to support this common good." But if we had acted back after opening SSL with a little more commitment.
I don't think we'd be at the place now where we have to worry about legislation coming in.
Petko: When I think about the industry, people always say, "It's a tech problem. The technical guys can always keep up." But if you think about it, even a media company, Disney, or Netflix, has software development shops internally. They're doing the same things as a tech company would or if you are producing a car in a factory.
That factory is updating its software every month or multiple times a month, that's also software. I think we tend to forget that when we start talking about software. It's not just the tech industry but all industries have software in them, even the banks have software. You name it, there's software everywhere, it's not a tech problem but it's a global problem.
Derek: I think one of the great things that have come out of this problem is not only are there cybersecurity vendors in this space. There are organizations like Google and Microsoft and JP Morgan. I'll name these companies like Target or Coca-Cola or John Deere or Disney. They may have developed some security analysis capabilities.
What they've determined is, I'm not trying to be a cybersecurity company. But we have this thing that we built internally to help solve this problem.
[32:50] Let’s Make SBOMs Available to Everyone
Derek: We're going to open-source it because everyone can benefit. We are not in the business to go and become a cybersecurity software company. So let's make this available to everyone. That has helped progress the state of the industry, which I think is very good. There are definitely a lot of good points coming out.
But again, when you think about just the baseline, do you have an open-source policy? Almost half the organizations do. Do you have SBOMs that you're using? Not even half of organizations do. There's still a lot of work to be done on the practice side. Even if the technology's available, it doesn't mean people are taking advantage of it.
It doesn't mean it's been instilled in the culture of these organizations to do the work or invest in making that happen.
Petko: I'm remembering back to when I used to write code. I never open-sourced it so it's a long time ago. The traditional software method is you test it as you're developing it, but most of the time you didn't really do testing until the very end. Of course, to the timelines, always get the software out, let's take testing from a month down to one day. I feel like SBOMs get put in the same bucket where you do it at the very end.
Maybe you get a subset of it. How many organizations actually use automation to do SBOMs?
Derek: If you're not automating it, there's no way you will keep up with the pace of modern software development. Even a best practice could be looking at SBOMs all the way through the development life cycle but it's most important to do it before release.
In the Line of Sight for Adversaries
Derek: Because once it's out there, you want to know that what you're putting out there could be in the line of sight for adversaries. Secondly, we were discussing with Log4J. If you put something out there and don't know what it was or where it went. The first two questions you ask are, "Okay. Today, a new Log4J vulnerability was announced, did we ever use it? If so, where?"
You can't answer that. That is a big problem for those in the industry. So, we do have to build it in more. It has to be automated just from the standpoint of going back and doing the math.
Let's say there were 3 trillion open-source component downloads in the last year, and there are 70 million developers. That is a lot of open-source components and packages per developer. There's no way you can manually approach this. I remember it was funny. So this was ages ago, I went to a very large insurance company in the United States. We were at Sonatype. They were looking at buying our solution at that time. We asked them, "How did you determine financially whether this kind of solution was going to make sense or not?"
They said, "When we were doing the math, it was either invest in this enterprise-level software."
I don't want to get into the pricing of anyone, but enterprise-level software is like six-figure purchases. Maybe, if you're gigantic, a seven-figure purchase. They said in order to get enough people to assess what we were doing, six, or seven years ago, we would need to construct a new building on campus. That’s to house all of the people to manually assess this.
Looking at Open-Source SBOMs
Derek: It wasn't fathomable to:
- Build a building
- Hire all those people
- Train them
- Keep up with the pace of development.
It was just impossible so, the math for them was really easy. It's going to cost a lot less to buy some solution to help us or even look at open-source SBOM capabilities that are out there. Analysis capabilities that are out there from OWASP or other organizations to bring those in versus the new building cost. I'm not in real estate. I don't know what a new building costs but it's more than seven figures I imagine.
Petko: We had you here two years ago and as you pointed not much has changed. But hopefully, Securing Open Source Software Act changes a few things. Looking forward to having you back to help us see how it's changed since then.
Derek: We're making progress because we're talking about it. There are good things happening across the industry and investment is being made. The Secure Software Supply Chain Act is yet another element of action that we can see. The OpenSSF 10-point mobilization plan is another one. I think that the things that I would continue to emphasize are 1) yes, we do need to put practices and culture in place but 2), what are your adversaries doing?
Because if you're not investing toward the capabilities of your adversaries, if you're twice as good because you implemented new practices but your adversaries are still 10 times better than you, you are still failing.
I think that the other thing that we need to look at, this would be a whole other episode.
The Cost of Cybersecurity Insurance
Derek: Maybe you guys can go and find a guest who talks about cybersecurity insurance and its impact of that on an organization. Part of why I think the industry is sometimes failing to act is the cost of cybersecurity insurance has risen. It has gone up substantially. If you look at the cost of that versus the cost of the breach or dealing with attacks, the insurance is still cheaper. I'm not as motivated to act in things like the Secure Software Supply Chain Act.
One of the areas where maybe it falls short is, let's say I don't produce an SBOM. What's my liability for not doing this or not fixing something that's in the SBOM? What is the cost to my organization and is there any enforcement around this? I think that's where we still need to make more progress on how we enforce it and the cost of not enforcing it.
Petko: You remind me of something. I've had this conversation just last week with large companies that are dealing with cyber insurance. A lot of them said the cost has gotten so high that I'm just self-insuring now. They said when they dug into the insurance policy, they're finding out the scope had changed. They're not covering as much as they used to. So, for those of you that do have cyber insurance, check your policy, don't just pay the check.
Derek: Yes, that's an entire episode that I am fully not qualified to be going to, but I know costs are increasing there. You look at some of the fines that become public on these companies and what part of those fines might have been covered by insurance.
The Self-Insurance Route
Derek: It doesn't feel like it's expensive enough for people to either go the self-insurance route or something else. But it feels like that's also part of the equation that we need to rectify.
Rachael: We have a lot of folks looking to come into the industry and make their mark somewhere. How did you get on this path of DevSecOps and securing software supply chains? We've talked to some people who have a Ph.D. in medieval history and now they're part of OWASP. It's just so fascinating the pathways of how people get to where they are. I would love it if you could share some of that with our listeners.
Derek: Prior to Sonatype, I was not involved with cybersecurity whatsoever. When I went into Sonatype, one of the fascinating things I was able to get visibility to right away was what was happening with Maven Central. It’s where all these Java open-source components were. We could analyze that data internally and when we were talking about this internally, how much was happening? How many open-source components, various organizations were downloading when we knew what percentage of those were good or bad?
I said we need to expose what was happening in this. It led to the state of the software supply chain report that I led the publication of for years there. But it was really just about being a data geek. I had access to all of this really cool data that had a story that was very relevant to people. On the other side of it, I'll give due credit to John Willis, one of the earliest advocates of the DevOps movement.
[43:20] Build Quality in SBOMs
Derek: When he was on stage continuously at DevOps events, he said, "You have to pay attention to Deming. He was trying to teach the manufacturing industry about “build quality in,” and how important that is to the future and competitiveness of an industry." I was able to marry a lot of the concepts that John Willis was talking about on stage and did a lot of reading on Deming myself.
Deming was teaching physical supply chain manufacturers and applying that back into software manufacturing and DevOps. It was the marriage of this data view of things where I was geeking out.
Understanding that the problem we were trying to solve in software and DevOps was very similar to a problem that other industries had solved. We weren't coming at this as new. We’re just trying to mimic what had been achieved in other industries which said, you actually can get better at building quality.
It can impact whole industries and the competitiveness and customer satisfaction tied to those products. Here was a new industry. I've worked in software for 30 years now. When it was about making software better, I was like, "Let's double down my efforts on trying to dive in. Let’s help make a difference here."
Rachael: See something, say something. Thanks again for joining us. It's been so great to have you back. It is disappointing we haven't made more progress forward. But it's exciting to see some forcing functions that are bringing this to the top of the list.
Derek: I appreciate it. The conversation is always lively here. Always look forward to listening to your latest episodes coming out as well.
Rachael: To our listeners, thanks for joining us this week. We look forward to catching you next time. Be safe.
About Our Guest
Derek E. Weeks is the world’s foremost researcher on the topic of DevSecOps and securing software supply chains. Over his career, Derek has held executive and senior leadership roles at The Linux Foundation, Sonatype, OpenText, and Hewlett-Packard. From 2014 - 2020, he championed the research and publication of the annual State of the Software Supply Chain Report and the DevSecOps Community Survey. Derek is also the co-founder of All Day DevOps, an online community of 100,000 IT professionals. In 2018, Derek was recognized by DevOps.com as the “Best DevOps Evangelist” for his work in the community.