Get a Break from the Chaos of RSA and Meet with Forcepoint at the St. Regis.


The Crisis CIO: Lessons Learned with Dave McDonald, Part 2

The Crisis CIO: Lessons Learned with Dave McDonald, Part 2

Dave discusses what we have learned from the COVID19 crisis and how it will better prepare us for future crisis.

Table of Contents

  • [01:24] Lessons Learned and Recommendations in the Time of COVID19
  • [07:12] Lessons Learned From Robust War Games and Drills
  • [14:26] Lessons Learned That Helped in Dealing With Adversary
  • [22:43] A Significant Change Based on Lessons Learned From COVID19
  • About Our Guest

Lessons Learned and Recommendations in the Time of COVID19

Dave: Hey, greetings to everybody. I'm amazed you asked me back for a second session. We rambled around on a lot of good stuff but I really enjoy the interaction with your team, Eric.

Eric: You got the juices flowing and you're near the ocean. I mean, how could we not have you back in?

Arika: Yes, we're hoping that when all COVID's over, it means you'll invite us out to Hawaii, so that's strategically why we invited you back.

Dave: You guys are keeping me from my race down to Ala Moana Beach Park and sneak into the ocean to get my swim in. I think we were talking in the last session about work-life balance. I need to take some of my own advice there. But really happy to be back with you.

Eric: I don't feel guilty about holding you back from the beach right now as I'm stuck in the middle of Maryland. Anyway, we wanted to have you back, talk a little bit about lessons learned, recommendations. We're in the time of COVID-19. What have you seen so far? What are your recommendations? Obviously nobody had the crystal ball, but now we kind of know how we've been impacted. What would you do differently?

Dave: Yes, so it's an interesting way to come at the question, sort of the what would you do or what would you have done differently?

Lessons Learned to Help Us Prepare for the Future

Eric: Yes, if we knew. Nobody could have known. We're not blaming anybody, but if you knew. Now that we know, how do we prepare for the future? What do we do differently?

Dave: So I guess a couple of things on that. That's a great way to start is ponder, look in the rearview mirror and say, "Man, what did I miss? What are those things we could have been doing to be vastly better prepared for something as extreme, and weird, and complex as this?" So I'll get at a couple of details, and some of these at the risk of being a bit obvious.

Dave: In our culture, I won't speak for everybody's culture, but I think I'm fairly safe in saying across large parts of the DOD and the government world, we've had a tendency to kind of hand wave at COOP, right? Continuity of Operations.

Dave: At the risk of being a little bit self-revealing, I don't think anybody would really have much of surprise with this. COOP is one of those things when IGs come to town, when the Defense Threat Reduction Agency comes to town, they say, "Hey, show us your COOP plan." It's one of those required staff mandatory compliance things.

Dave: Culturally, we've largely treated it like that. I am certain there are exceptions out there. There are organizations that, man, they really put some thought and some muscle, and they put their money where their mouth is, and their COOP plan is ready to go. I would suggest that I have real severe doubts that anybody's COOP plan factored this crisis in.

The Lessons Learned Can Be Hand-Waved, Cynical Exercise

Arika: And if they did, I want them to play the lottery with me.

Eric: You only get that one right once in a million years.

Dave: If that savvy executive out there who figured out that dimension of the COOP plan is around somewhere, hire them. Because they really had a crystal ball. So I think, at the risk of sounding like I'm dancing around the question. So COOP planning and resilience planning, the lesson learned is it can't be a hand-waved, brushed-away, cynical exercise.

Eric: Take it seriously. Do your drills. Rehearse. Learn from it and then enact improvements.

Dave: And I use the phrase put your money where your mouth is, right? And this is in no way intended to sound like an, "I told you so," or, "Man, I saw this coming," because I didn't. I had the same kinds of blind spots that a lot of folks had.

Dave: But I will say this, for the better part of 10 years in my current position, I and my staff, our whole team, the great bosses that I've worked with, we've had some fairly deep, nuanced, and thoughtful discussions about resilience, about what's the worst scenarios that could occur as we should, right? We're a war-fighting organization, right? But we thought in terms of near-peer or peer adversaries and getting attacked in our [inaudible 00:05:14] fixed fortifications.

Dave: What would the start to a war look like, both from a cyber perspective as well as a kinetic perspective? Attacks on infrastructure, getting isolated in terms of telecommunications connectivity. So we have had some very deep and thoughtful thought drills, experiments, and what if, and what now types of thinking on resilience.

Lessons Learned From Robust War Games and Drills

Dave: This is where it tends to fall apart though, is from a resilience and COOP perspective, a key dimension to that is what are you willing to pay for? Are you really willing to CapEx and OpEx the resilience margin, the COOP margin that you're going to need when that worst-case scenario starts to unfold on you? That's where it gets tough.

Eric: So Dave, what would you give up? I mean, that's one way. If you're not getting any additional funding in the navy, we can give up ships, we can give up aircraft, we can give up personnel, or just at the IT level, what would you have given up? Let's assume the funding stayed the same and you knew this was coming. How do you do that balance exercise?

Dave: I won't entirely evade the question, but that's why we pay our senior three and four-star and really senior executive Department of Defense, Department of Navy Civilians. That's why we pay them the big bucks, it’s to have those conclaves, to take the readouts from the National Defense Strategy and maritime strategy of the United States, and look at the current readiness and state of the force.

Dave: And then take the lessons learned from some of the really robust war games and drills that we've done and then shape that as a balanced investment portfolio. And again, in the telecommunications, IT, and command and control services, the things that CIOs are traditionally going to care about, in the equation of resilience and crisis response, crisis margin, maneuvering margins in a really tough, almost unimaginable scenario, that's a balanced investment decision on the front end.

Lessons Learned by Looking in the Rear View Mirror

Dave: Eric, you started with the question of what should we have done or looking in the rearview mirror, how could we have better prepared? It really does all start with deep, thoughtful, complete strategy development capabilities analysis, architecture analysis, risk and resilience margin analysis. And then put your money where your mouth is, right? You cannot just hand-wave at this stuff. It's kind of interesting, I'll give you a data point without going too far with it. I mentioned the Defense Threat Reduction Agency, the DTRA.

Dave: So, this is their stock in trade is to go around the defense and national security community and look for worst-case what-ifs and what nows. And then they'll produce a 40 or 50-page report that suggests all the things that you really need to be thinking about. And A lot of it takes the form of requirements and investment categories.

Dave: You need to build risk and resilience margin into your infrastructure, into your IT capabilities, into your facilities, in your distribution. And you need to think real hard about your people because your people, as we're learning in this crisis, are every bit if not more the critical asset as our facilities, our systems, our stuff.

Dave: Well, I think I said in our previous session, I think a lot of CIOs in this particular scenario are finding they're having to step up as kind of the chief resilience officer. So it ain't just information and it ain't just systems. It's where are all the people, who are all the people, what do they know, what are they acculturated to?

Knowledge Management Design in Runtime

Dave: Are they ready for this in some form or fashion? How quickly can we train them and get them ready to operate and think this way? So the crisis CIO or the resilience officers got to think people, process, tools, culture, runtime governance. I have a lot of thoughts to offer about knowledge management design in runtime like we're doing right now, especially as top-down solutions are being mandated.

Dave: And then the backplane of all this is don't forget cybersecurity and compliance. So there's a lot of dimensions for a crisis CIO as a chief resilience officer to think across people, processes, tools, culture, and leadership.

Arika: So Dave, just to that point, I think right now Eric's question is what would you have done differently, how would you balance resources, things like that. But from a practice standpoint, once this is all over and this applies to government, whether you're a defense agency or healthcare, private industry, what happens from a practical standpoint next, from your perspective as a CIO? I mean, next year do you kick off a meeting where you just overhaul everything? Or what advice would you give for someone who's thinking about this from a very, I would say now, as from a future perspective?

Dave: So Arika, that's a great question. I need to sort of caveat or frame the answer upfront. And that is there's a lot of days I come into work and I think, "I wish I was one of those CIOs who kind of had the whole portfolio”.

The Ear of the CEO and the Ear of the CFO

Dave: The ear of the CEO and the ear of the CFO, and ultimately my team and our extended partners, we're collectively making the decisions on technologies, enterprises, what type of cloud configurations? What type of collaboration or unified communication solutions, business solutions, workflow solutions?

Dave: So the caveat is this; I'm not one of those kinds of CIOs. I'm an operating CIO who takes a heap of top-down decision-making and then tries to figure out how to rationalize it, implement it, and achieve the operational value of it.

Dave: In that context, what I will do and really have done and what my staff, my team does a lot of is we're a real squeaky wheel. We're at the table as a requirements generator and something of a technical and operational advisor on many of the acquisition decisions that get made at the Department of the Navy level.

Dave: With our PEOs and our program management offices, the technical directors and we even occasionally get a voice at the joint table, at the DOD CIO, and the DOD Acquisition level. So we're a squeaky wheel, we're vocal. I think we're well-informed and we try to be rigorous about that. We'll continue to do that. And I think on the front lines of this COVID-19 crisis response and some of the things that we're thinking about in the vein of crisis CIO and get the runtime decisions right.

Dave: I think in the coming days, Arika, we'll have plenty to offer the acquisition organizations in terms of ideas about what's most important and what technology selection, design criteria, governance dynamics, or governance structure is going to be most effective of course going forward.

Lessons Learned That Helped in Dealing With Adversary

Dave: So I'm not a buyer of technology. I'm a fielder user and implementer on the operational end of technology, but the good news is we're at the table with the program offices that are going to figure this out for us in the coming years.

Eric: Dave, you have a large percentage of your workforce working from home now. How are you protecting them, the intellectual property from ransomware? We've seen the adversaries really ramp up the attacks on corporate personnel working from home, right? The adversary's very industrious. They adapt very quickly. How are you dealing with that in this time of crisis?

Dave: Eric, there's a couple of I think really straightforward short answers to that. Number one, we're following the rules. This is a case where I'm really thankful that the guidance, the rules, the directives coming from the DOD CIO, coming from the DON CIO, Mr. Aaron Weiss and his staff, coming from our chain of command has been really crisp, really precise, and really quick.

Dave: So they're really paying attention to that idea that lives in dynamic knowledge management of if you let chaos and entropy take hold at will, right? Because everybody's got an idea about it and they're going to run off in lots of different directions. And many of those directions are not likely to be cyber secure at all.

Eric: So you're retaining that structure, if you will.

Dave: I think it's a reasonably fair balance between hey, collaborators got to collaborate, right? Organizations have to continue their operations.

Choose Your Own Device Compliance Model

Dave: They have their operating cultures, but we're going to do it in a certain governed, managed, directed way and nobody's going to forget cyber security. Now, we have some advantages.

Dave: The DOD has had a reasonably good culture in training and mentoring cyber hygiene where it runs a little bit into a risk area is all of these personal devices. I'm sitting here on a $350 HP laptop and you got others on iPhones, droid phones, all manner of foreign rapidly-introduced personally-owned devices with plugin, CAC readers, and so forth. So, I think one of the epiphanies that's really going to pop out of this quick is this notion of either bring your own device or choose your own device.

Dave: I had an occasion this past week to post a quick blog on this on LinkedIn. I've always really liked a choose-your-own-device governed, enforced compliance model across some more distributed mobile take-home, go-home type workforce. What I like about choose-your-own-device over bring-your-own-device is it could strike that balance between agility, maneuverability of choice, but also have, I'll call it sort of a comply-to-connect and a zero-trust enforcement.

Eric: You have the control element that you need as a CIO.

Dave: Yes, sir. That's right. And if I had to guess, you watch as the dust starts to settle on this thing, by summer or fall, there's going to be a lot of chatter in the DOD press about choose-your-own-device and comply-to-connect, and how to extend your zero-trust architecture boundary to encapsulate a much more robust endpoint community.

Eric: Arika is the expert of the millennial generation. What do you think about that?

The Rise in Mobile Workforce Due to Lessons Learned From Present Situation

Arika: I think that makes a lot of sense. I mean, I think as Dave just said, "It will strike that balance," that I think especially younger generations, I mean, they like to have that sense of choice, but that still allows organizations to have that sense of control. So, I think it makes a lot of sense.

Arika: I think that it will be interesting just as a whole to see what the workforce in general looks like both from a security standpoint, but just from an overall standpoint. I think we're just going to see much more traditionally buttoned seats industries including government, which is very much some agencies over others align that way.

Arika: I think we're just going to see a much more mobile workforce because I think we've learned a lot through this situation that you can continue business operations even those that when it comes to things like security and such, you can do it as long as you have that $350 computer and a secure network.

Eric: So Dave, last question. We're noticing some of our engineers who have always worked from the office, they now want to work from home. They want to get back to the office but they want that latitude to work from home. Do you expect the navy and the military to allow a more liberal work-from-home process through the learnings here or it'll go back to the way it was?

Dave: I'll tell you Eric, that is a great question and I don't think it would surprise them.

Herding Cats Versus Letting Cats Roam

Dave: There's been some stuff written in the public press about the resistance over a period of really many years to teleworking not just by DOD but really by large segments of the government.

Eric: I was at the intel where they brought us all back into the office, so I get it.

Dave: So, I'm not sure you flip the switch on that culture overnight, although again, I've used the term epiphanies. I think a lot of folks are having epiphanies about a lot of things with respect to their people, with respect to trust, empowerment, the balance between chaos and design. Herding cats versus letting cats roam and do what they need to do.

Dave: And then how do you also assert appropriate security measures, OPSEC, and some governance. Those things don't just go away because the dynamics of a younger, free-er workforce are coming into the DOD. We're still the DOD. We're still the nation's warfighting organization. We're still the nation's intelligence capacity. So all of those more conservative risk issues still have to be part of the fabric of our culture and the substance of our governance.

Dave: So, I don't think this changes overnight. I think the epiphany is though people are having, and again I mentioned this; choose-your-own-device. Part of that is the enterprise buys that device for you. I don't bring my personally-owned $350 HP laptop to the architecture.

Dave: The enterprise considers the broad benefits of a mobile, agile, maneuverable, resilient workforce. Think of any other types of scenarios that would have forced large numbers of our expert workforce somewhere else.

A Significant Change Based on Lessons Learned From COVID19

Dave: These natural disasters, attacks, the beginning of a war where a peer adversary decides to level set the battlespace by going after critical facilities.

Dave: We immediately have to redistribute large portions of our expert workforce and we don't have a lot of huge margin in terms of that expertise. It's at capacity anyway. So this idea of imagining the scenarios where that investment in resilience margin and investment in a distributed workforce and getting everybody used to being able to operate seamlessly that way.

Dave: That is a very important strategic investment to make. I think the guys and gals who think through architecture technologies and governance are going to have to pay attention here. I think distributed endpoints, mobile-governed, compliant, zero-trust endpoints purchased for the critical workforce that would have to exercise such a scenario. I think that's good thinking here and maybe it took COVID-19 to get us starting down that path.

Eric: I think we'll see a significant change based on this.

Arika: Well, thank you, Dave for joining us again on the podcast. It's been great.

Arika: We really, really appreciate just your thoughts or insight. I think, hopefully, in terms of us really using lessons learned and best practices, the best is yet to come. So, let's all hope so.

Coming Together as a Team

Dave: The country's going to get over this thing. I think so far from what I've seen, I mean, this really all started escalating quickly that first week in March. I had just come off travel from both Australia and San Diego. So far what I've seen is the nation should and can be proud of its defense department, its military forces, and the different support organizations that are part of that ecosystem.

Dave: The American public should really be proud of what folks are doing and how they have responded to this and stayed on the job, focused on the essential, come together as a team. We haven't taken our eye off the ball. We're making smart decisions on the fly, very quickly. A lot of wise senior leaders really paying attention here and giving us all good leadership and guidance.

Dave: So, I think Americans should be really proud of what their defense department and their military forces are doing right now to respond and be supportive in this over and above just the obvious stuff, which is things like the USNS shift over to New York and Los Angeles, and so forth.

About Our Guest

David McDonald, Navy Telecommunications, Information Technology and Cyber Operations. Navy and Intelligence Community professional with dual career tracks as a DoN civil service leader and manager and a uniformed Navy Reserve senior officer (now on the USNR retired list). I bring 37 years of varied professional experience to the table, with targeted expertise in strategic planning/execution, program management, project management, organizational design, professional development, mentorship and aligning business/programmatic solutions with mission operations accomplishment.

I'm a cyber professional, in the "lingo" of today, and a trained/certified, warfare qualified and operationally experienced Naval Cryptologist and Information Warfare Officer at the core. Career path has been fortunate, sometimes downright lucky, taking me on a path from tactical Cryptologic operations on submarines during the Cold War to major SIGINT field station operations, space systems operations in the National Technical Means (NTM) community, submarine programs management, military intelligence operations at the Joint Combatant Command level, and, ultimately telecommunications and IT program management.

I have sub-specialized in Defense acquisition program/project management, architecture planning and systems engineering along the way, and have complementary sub-specialities in Knowledge Management, Contracts Management, Training Program and Systems Development, Systems Architecture Development, and a few other things that I can dust off if the situation warrants. I'm from the school of Servant Leadership - have read and taught the literature and practice, and believe in creating organizations that are organic, defy usual bureaucratic "wire diagram" boundaries, and foster innovation, collaboration, continuous improvement and teamwork. If one has been fortunate enough to be mentored through a long career, one then must mentor, with selflessness and dedication.

Listen and subscribe on your favorite platform