Derek Weeks is a huge advocate of applying proven supply chain management principles into DevSecOps practices. It’s designed to improve efficiencies, reduce security risks, and sustain long-lasting competitive advantages.

Table of Contents

• [00:43] All Day DevSecOps
• [06:44] Enlisting People Who Help Build the Tracks
• [14:28] The Latest Efforts in DevSecOps
• [23:04] Percentage of Organizations Using DevSecOps
• [27:24] The Cultural Barriers to Transformation
• About Our Guest

All Day DevOps

Carolyn: This morning, we have Derek Weeks, Vice President and DevSecOps advocate for Sonatype. He's also the co-founder of All Day DevOps, which is an online community. It’s also a 24 hour online conference that over the years has attracted over 130,000 participants globally. It just blows my mind.

Eric: In over five years, Carolyn. We're not talking for two decades. We're talking relatively recently. It's huge.

Eric: I have an admission before we get started. I worked in R&D while I was still in college. And I was a software tester, Derek. I know nothing about this topic, or very little. Because I made it about two weeks and I self-selected myself off the island.

Eric: I couldn't stand looking at three different computer monitors, multiple computers running test algorithms, and not talking to anybody. I’m not a developer, and you're going to school me today.

Derek: This is a pretty easy topic. You can get into the complexities of it, but at a high level it's something everyone can understand. The need for and the purpose of what we're doing in the DevSecOps realms is managing software supply chains among others.

Eric: I can't imagine it being easy. Development, security and operations, it's just easy.

Carolyn: What does DevSecOps stand for? I want to talk about this 24 hour conference that you've been doing for five years online. Because you want to be online, not because you have to be online. Tell us about it.

There’s Enough Momentum in This Space

Derek: One of my friends in the community said we were doing online conferences before it was cool to do online conferences. We started this all day DevOps five years ago. It really grew out of this experience that me and Mark Miller, the other co-founder, were having.

Derek: We went to, 20, 25 DevOps security developer conferences throughout a year. We’d meet three people from Wells Fargo, five people from DuPont, one person from Intuit. We're talking to them about what they do in DevOps and their teams back home at the office. These groups could have anywhere from 10, 20, 200, to 4,000 DevOps professionals in their organizations.

Derek: Not all of them could attend a conference. I said, "Mark, I think there's enough momentum in this space of people wanting to learn about DevOps. We could invite all of our friends who we know who speak at conferences. Pull together a conference and run it online, make it free, make it available for everyone."

Derek: That first year, five years ago, we had 13,000 people participate in this. The first conference only had 57 speakers. This year on November 12th, we'll run 24 hours. We'll have five to six simultaneous tracks, we have over 180 speakers. We will have a government track as part of the conference.

Derek: We also have a DevSecOps track as part of the conference. A couple of things that make this experience unique, one, it's free for anyone to participate. We always say, register yourself and register your team. Two, we provide a Slack workspace for conversations. That's really important because the conversations in the community are really what you learn from.

Creating a Community Experience

Derek: You can talk to every speaker at the conference, ask them questions. Even though the conference goes away after November 12th, all the sessions are recorded online, the Slack workspace stays persistent. If you want, go and talk to those people at any time after the conference. There are people talking on Slack today in our workspace there. It creates this community experience for people to participate.

Derek: Another thing that's really important about this conference is, we don't allow any vendor pitches as part of the conference. When you're coming, you're hearing from thought leaders. You're hearing from PA practitioners. You won’t hear me talk about the awesome products we have at Sonatype for an hour as you're showing up.

Derek: You’re not hearing 20, 30 other vendors pitching their product. It's really meant for practitioners and leaders to get information about what's happening in DevOps, and the latest practices and to share that with their peers across the world.

Eric: How do you pick the topics, the tracks, and get the speakers and get everybody going? How do you structure it?

Derek: It takes a community to do this. Right from the very beginning it was not, well, Mark and I came up with the idea. It was a group of people from the community. We had some folks from Oracle, some folks from Splunk who came in early on. I'm forgetting the companies where these guys worked originally, they're all now at a company called Verica.

Derek: We had three friends in Austin, James, Ernest, and Karthik, who came in along with a couple of others from Intuit and other companies.

Enlisting People Who Help Build the Tracks

Derek: They said, "If we're going to do this, and we're going to cover DevOps, we should talk about different topics that make sense." We evolved the tracks with community input that way. We have continuous delivery, we have cultural transformation, we have government, we have a security track.

Derek: We have modern infrastructure, like cloud, Kubernetes, and that kind of thing. Mark and I are not the ones organizing the track. We've enlisted people throughout the community that help us build the tracks.

Derek: They identify speakers that we should have on the agenda. But we also do a big call for papers. We had hundreds of submissions this year for the positions that we have available.

Carolyn: Do you run it through Sonatype, or this is all just you and your friends, and the community?

Derek: Sonatype is always a sponsor of the conference, and has been from year one. There are other sponsors like CloudBees that have been vendor sponsors from year one. NowSecure is another vendor sponsor that we have. Google is also a sponsor this year. We also have corporate sponsors. Northrop Grumman is a huge sponsor for the conference this year

Derek: They’re supporting our government track all through the 24 hours. We also have State Farm as a big conference supporter/sponsor this year. And hundred community groups throughout the world help us spread the word about the conference each year. It's a pretty amazing experience.

Derek: November 12th, all day. We start at 3:00 AM Eastern time, and we end on November 13th at 3:00 AM Eastern time.

Carolyn: You're on the entire time?

Understanding What DevSecOps Is

Derek: I stay up 24 hours. This is the one time a year that I do it. I always think this is about the craziest thing I could do, but it's so fun when we're into it. There's a lot of adrenaline flowing. Each session is about a half an hour, so every half an hour we're changing with new topics, new speakers.

Derek: People are logging in from different time zones around the world and sharing. They're posting selfies on Slack. Here I am in Belarus or Paris or Argentina, or wherever they might be around the world. So it's pretty cool to see all the excitement buildup.

Eric: How do you sign up? Google All Day DevOps?

Derek: alldaydevops.com, or just Google All Day DevOps. It's free, so you can sign up as many people as you want.

Eric: I don't see myself spending 24 hours on the 12th, based on my background and interest. You heard my story about being a software test engineer. But I would love to spend some time learning about DevSecOps. It's an area that I don't understand well.

Carolyn: I'm fascinated by the logistics of this thing. DevSecOps is a term that gets tossed around a lot in my organization. I don't really get it. Can you talk to me like I’m a fourth grader and explain what DevSecOps is, particularly in the federal space?

Derek: The evolution of development and even security has continued to change over the years that we've had these practices. But DevOps, at its core is, one, a mix of cultural practices within an organization. They’re supported by technology practices as well. What it really tries to get to is fast feedback loops.

The Fast Feedback Loops

Detek: The fast feedback loops are, if I'm doing something, and if I make a change in code how quickly do I know whether that is a successful change, or whether it's a problematic change in the code? When I started off in software development decades ago, we built a piece of software.

Derek: We shipped that piece of software once a year, or once every year and a half. When things went wrong with that release, you would have to either offer hotfixes, patch updates somewhere. But those sometimes took a couple of months to get out the door. The feedback loops were really slow. We'd spend a year developing the software.

Derek: If we didn't have the feature that you wanted, you’d wait another year or year and a half to get that. Then if there was a bug, maybe you were lucky to get the hotfix out there. That feedback loop was really long. What people figured out is, you could actually release bits and pieces of new functionality and new capability.

Derek: Release those quickly, and if they work, then start moving on your next thing. If they don't work, then you’d know. If it took you a minute to release this, you could pull it back in a minute. You could fix it and then send it out again. You’re able to release new capabilities and new value to your customers using that software faster.

Derek: DevSecOps brings security into that fold. Normally in a development life cycle, you would build the software. Then you would have a security practice bolted on to the end of the software development process.

The Feedback Loop is Inefficient

Derek: You would do the security check there, and the security team would then give that feedback to the developers. In the feedback loop sometimes there were two weeks, four weeks, a couple months depending on the organization.

Eric: Meaning you wrote something, I inspected it. Two, three, four weeks later, you get something back on what you wrote. Now, you've got to figure out, what was I doing then? How do I get back in the process?

Derek: Right. So you can imagine with this podcast, we recorded it today. Some editor a month from now is reviewing it and saying, "Eric, we really messed up on this part. Can we go and re-record?" And you're like, "A month ago? What was I doing? Who was I talking to?"

Derek: So that feedback loop is inefficient. And so you need to think about, how do we take security and embed it earlier in the development life cycle? Some of that is tooling, some of that is information. But you can imagine, a developer is writing a line of code.

Derek:  As they're writing it, there's something effective to a spell check built into their code. It says, hey, there's a security error flaw that you're introducing here as you're coding. You might want to correct that. That feedback loop would be instantaneous.

Eric: How do you do that?

Derek: That is a thing. It comes in all different flavors. Say you're pulling down a Docker container with an application that you want to use in it, can you flag that immediately to say, I downloaded it. I've scanned that, I know there's vulnerability in it. I know immediately when I want to use that.

The Latest Efforts in DevSecOps

Derek: The open source piece of code that you're borrowing to help construct your application, is that safe or not safe to use in your environment? There are some types of various static analysis tools that will look at your code as you're writing it. It can give you that feedback. It's amazing when you see it in practice. One of my favorite stories was in Washington, DC. I was presenting to an OWASP meetup, so that's the Open Web Application Security Project.

Derek: I was telling them about some of the latest efforts in DevSecOps. One person in the audience was a developer. He said, "I just got an alert from GitHub where I submitted my new code. It says there's a security vulnerability, and I have to go and fix it." This was a developer using GitHub, a developer tool. That tool had information supplied back to the developer after they submitted their code, that there was a problem.

Derek: They’re then able to go and fix it. This didn't require a security team reviewing it. It didn't require a review of six weeks later of their code to come in. But the developer has information at their fingertips seconds after they're committing code to their code repositories. There are continuous integration tools, or other development oriented tools that they're using.

Carolyn: Are these tools standard, or does every organization have different processes, different tools

Derek: There are a number of different standard development tools that people using. Different vendors in these spaces are using a development tool that’s standard for developers who just want to code. Or they’re using a security tool that's figuring out how to feed or surface information about security to developers inside their own tools.

What Devsecops Is Bringing to the Table

Derek: It's this construct of the Word doc and spell check. Spell check is actually a separate application, but it's feeding spelling information to the person writing the doc. You can see that construct, you don't have to send your doc off to an editor. You're just getting that information surfaced up so you can make intelligent, fast decisions. Fast feedback loops to get that flaw corrected in the code.

Derek: That's really, in essence, what DevSecOps practices are leaning toward. How do we get information to the developers faster about their code and what's going on so that they can take corrective action? Also, it works in operations. Once the application is deployed into production and a security flaw is found, how quickly can we surface the feedback loop to say a flaw has been found?

Derek: Take action on it before adversaries can go and attack that vulnerability within the application. Think about it from a standpoint of feedback loops and how long or short are those. Then it can help you better imagine what DevSecOps is bringing to the table.

Eric: Carolyn, 20, almost 30 years ago, that would have been me. Some knuckleheads in college are trying to look for those vulnerabilities. Doing that and running iteration after iteration in the visual test suite by Microsoft, or Rational back in the day. It was a horribly inefficient process. Derek question for you then, that used to take months to release code.

Eric: The place I worked, we had an 18 month release cycle. We were trying to compress down to 12. That was the big goal back then. How do you release code today, and how quickly?

For the Best in Breed

Eric: You’re still retaining some level of, you need release notes, you need some consistency. You can't release code hourly. How does that work?

Derek: Actually, you can release it incredibly fast.

Eric: How do you do it reliably?

Derek: In these new environments, especially for the best in breed out there, Amazon or a Netflix, or a Google, they're at tens of thousands to hundreds of thousands of releases every day. The way that you do this at scale, and super fast like that, one of the things that they've figured out is that a lot of the testing that was done by individuals could be codified.

Derek: So if you'd say, I'm running a test to look for this kind of thing, you could put that test into code. Instead of a quality assurance person saying, I just got the code from development. Now I'm going to develop my 20 tasks that I run against this code. I'm going to figure out what works, what doesn't work. Then I'm going to send that information back to development.

Derek: The QA people, or security people, or performance testing, or unit testing folks with that expertise. They said, “I'm actually going to spend my time not running individual tests for these applications. I'm going to put these tests into code. So every time the code is released into the system where it's integrated, I can run this test automatically against that.”

Derek: People went from being able to manually doing, if you’re doing five tests before manually against this code, you'd say I codified those five tests. Now I can actually spend my time codifying five more tests, and codifying a hundred more tests.

Automating Through DevOps Practices

Derek: You can release or submit software for testing during this process that might have thousand tests run against it within seconds. It allows that code to not only be developed, but be tested, and understand that the quality is working before it's going out. A lot of information about whether the code is high quality or not and meets the standards of the organization.

Derek: They have been automated through these kinds of DevOps practices. In fact, QA, for the most part, in this DevOps environment, has effectively gone away from a personnel structure. Because the testing has been automated and embedded in. It's not that the function of QA has disappeared, the person doing manual tests has disappeared. The person creating automated tests is still there.

Derek: He’s figuring out. There's actually no end to the number of tests that I can create and run against this code. As long as running those tests, isn't slowing down the development process and you're getting the fast feedback loops there.

Eric: I want to make a joke that I was ahead of my time. But the reality was, I was really, really bad at this. I wanted to talk, I wanted to see people and interact. Just look at screens all day long. I love the fact of building it in and automating it.

Carolyn: Is it released? If an error is found in the code, it's automatically released out to the customer, or we wait until the next shipment?

Derek: The way that the feedback loops work is, you're at step A, and you're going to step B. When the code goes to step B, we do a check.

Percentage of Organizations Using DevSecOps

Derek: If it passes all those checks, you go to step C. When it doesn't pass, the automation takes the code back to step A and says, we noticed a problem at step B. It needs to be addressed. Let's address it and then send it back to step B, see if it passes, if so, go to step C. It's not just one check at the end of like, is the code ready?

Derek: It's literally, there might be 100 steps that all have feedback loops built-in. If you get to step 53 and you get an error, it sends you back to step 52. Or it sends you to the appropriate step and says, we spotted an error. It's hopefully something simple because there was a little change.

Derek: Can we correct this and then get it to move on? If you can do that quickly enough, a developer's going to appreciate that. Kind of, I just committed a code a minute ago, I'm getting an error. Let me correct that error on the new bit of code I just submitted. If I can get that within a minute, I'm happy to work on it. But if I got that three weeks later, I'm not really happy to go back. I touch what line of code when?

Eric: Especially if it's 2:00 AM and there's nobody checking anything at that point. You can keep working when you're in your groove, as opposed to this artificial hard stop. So what percentage of organizations are using DevSecOps? I guess that's the way to position it, as a development methodology these days?

Derek: We've seen DevSecOps evolve over the last seven years or so. I've certainly been looking at this space more intently.

One Critical Thing About DevSecOps

Derek: I do an annual DevSecOps community survey as part of that. It’s where I can see how the industry is investing in these new practices. In fact, for the 2020 State of the Software Supply Chain Report that we just released last month, this was August, 2020.

Derek: We did a survey of about 670 development teams. Within that, about 25% we would consider high performance DevSecOps practices. Within the different types of things they were doing within their development teams.

Eric: 25% of all companies or all respondents?

Derek: 25% of all respondents within this. That's basically per company that would be involved. It's not this brand new thing. No one's actually doing this. This is just visionary stuff. There are a lot of organizations doing it, and a lot of organizations attempting to get faster at it. Let me point out one thing that’s also critical about DevSecOps practices and the employee bases that you have.

Derek: Whether it's private sector or in government agencies, and why approach this. We've talked about the faster feedback loops and that makes work a little easier. It makes work more efficient. If you're a developer working in this kind of environment, where are you going to be happier?

Derek: Is it the one where you're getting faster feedback loops? Or the one that you're in this slow monolithic, release every year and a half kind of environment? The one where you're using newer tools and newer capabilities? Or the one that you're using old legacy tooling and capabilities?

Derek: What others have found in industry surveys around DevOps and DevSecOps is, those organizations adopting these practices generally have much happier employees.

Much Happier Employees

Derek: If you have much happier employees, they're going to be more engaged in their work. They're going to be doing higher quality work. They're also going to attract other people, those who want to work at these kinds of organizations to your own business.

Derek: Those agencies who want to work in that kind of environment. There’s not just a technological payoff or a customer benefit payoff that we can deliver more value or more features to market faster. If you have happier employees, that’s a huge competitive advantage.

Carolyn: Not that I want another government mandate. As you're talking about this, I'm thinking, why would we accept software from anybody that doesn't use these kinds of practices? This has got to improve the security a hundred fold.

Derek: It certainly does. But it's easy to say it, and it's harder to implement. Part of that harder to implement has to do with the culture of the organization. Let’s say I'm in a government agency and I'm running a QA team. You're telling me I'm no longer going to build all of these tests, manual tests for the software that goes out.

Derek: We're used to our release cycles where we're testing that bit of software once a year. We are planning around that cycle of planning once a year. Then you're going to tell me we're all changing our job descriptions to write tests to be automated. We're going to do lots of releases. I don't like change. I'm going to fight against that culturally.

Eric: Or if the QA team under a contract and you modernize, or you go this direction, that contract goes away.

The Cultural Barriers to Transformation

Derek: There's a lot of cultural barriers to this kind of transformation. In our all day DevOps conference, we have a whole track associated with this topic. Because that is the most challenging part of DevOps adoption. It’s how do you get the mindset to then work in this kind of environment? To map out what work needs done in this environment? And to get all the different teams bought into, we used to work this way, and now we're going to work this way?

Derek: Don’t have people feel like, my job's at risk. Is my job totally transforming? Am I safe here? If I go down this path, am I going to be better prepared in my future for the next job that I'm looking for whether it's at this agency or somewhere else? That's something that everyone has to keep in mind as the leader of an organization doing software development.

Carolyn: Do you see the federal government, are the agencies there embracing this at a faster or slower rate than commercial?

Eric: I'm going to go slower. Remember how I started the conversation. Derek?

Derek: It is a little slower, but it really depends. It depends on the place. There are some places that are making massive investments in DevSecOps or DevOps practices across the government. Department of Homeland Security, huge investments in this space. The IRS, Navy, Army, all the intelligence agencies are working in this space to improve what's going on.

Derek: I know, just locally in DC, the US patent office has fully embraced DevOps, and has for many years. They hosted the local DevOps Days in DC for many years.

Another Reason DevSecOps Comes Into Play

Derek: Until we outgrew the space that they had in their auditorium to host 500 people coming to DevOps Days locally. There is a lot of investment, but this really rings true for the private sector as well as government.

Derek: Not everyone in the private sector is adopting these practices. Some are struggling to adopt these practices. Some are threatened by competitors who are adopting these practices. There’s a need to move more quickly. If your competitors can deliver more value to the market faster, then you're in an advantage in the market.

Derek:  For government agencies, you have to look at, how are we delivering value to our constituents? To our stakeholders, our customers within the federal government.? Or perhaps be able to do it faster than what our adversaries are doing. That's another reason DevSecOps comes into play.

Derek: We can talk a little bit about what I found in the Software Supply Chain Report this year. In terms of what the adversaries are doing, they have us needing to stay on top of our game.

Carolyn: I just looked at the time. I thought, at best, this was going to go over my head and I would just have to nod a lot. This is not as boring as I thought it was going to be. I'm not a developer. But honestly, I would love to revisit. To come back to what you just mentioned in the state of the software supply chain report and to talk about what you found with adversaries.

Eric: We need a teaser. Give us something to hang onto.

A New Security Vulnerability

Derek: Here's the teaser. A lot of people in these DevOps practices are looking at, how do we release faster? If you were deploying once a year, and then you're like, man, we've improved things incredibly. Now we release once every two weeks, amazing. 24 times improvement in how fast we can release.

Derek: So if there is a security flaw in our code, we can fix that in the next release two weeks out. That's awesome. Now look at it from the adversary side of the business. A new security vulnerability comes out. The adversaries are scanning your network for that flaw within 24 hours.

Eric: They have a short window to find openings.

Derek: They can exploit that flaw in three days. If you can release software once every two weeks, but the adversary is going to exploit the vulnerability in three days, you're still too slow. You can't focus on these internal benchmarks of, we're 20 times faster than we were before. If your adversaries are even faster, you’re in trouble.

Carolyn: This has been fascinating. Thank you so much for joining us, Derek. I will get with your people to schedule you for our next episode. I'll look forward to part two with everyone.

Eric: All Day DevOps, 12, November, 3:00 AM to 3:00 AM. Is that Eastern?

Derek: Eastern time. On the program, you can pick your time zone. We'll convert that and tell you what time it is in your local city wherever you happen to live.

Eric: I'm glad you guys are there. It sounds like you're making the world a better place.

Have a guest you think would be great for the podcast? Please email Carolyn cford@forcepointgov.com

About Our Guest

 

Derek Weeks is a huge advocate of applying proven supply chain management principles into DevOps practices. To improve efficiencies, reduce security risks, and sustain long-lasting competitive advantages. He currently serves as vice president and DevOps advocate at Sonatype.

Derek is the co-founder of All Day DevOps, an amazing virtual conference bring together DevOps practitioners and thought leaders. It’s the largest virtual conference in the world. It’s educating DevOps professionals through online training and blog content.

Hosts over 180 local community events in 20 countries around the world. Since its founding in September 2016, our community has grown to over 130,000 strong.