close
October 14, 2021

Defending Against Ransomware: Cyber Security Insights with Paul LaValley

On this episode, guest Paul LaValley, Chief Information Officer at Yuba County, shares how his wide variety of work experience helps him and his team excel. Paul worked with several startups in the Silicon Valley and with his leadership, was able take 4 of them to the public stock market. As a CIO now, security is always on his mind. Listen for some tips and tricks to prevent future ransomware attacks and stay tuned for a chance to hear how his teams use resilience and motivation to push each other to overcome their goals.

Transcript

0

00;00;00;00 - 00;00;23;22

Narrator

We're in an era of rapid change where resilience is vital. The Davood for Thought podcast dives into the most important topics in government and technology today. Our host, Davood Ghods, sits down with his vast network of colleagues to dish on the tech challenges that affect us all. Follow this podcast on your favorite platform and join the conversation by sharing it on LinkedIn, Twitter, or Facebook.

 

00;00;23;24 - 00;00;48;17

Davood Ghods

Hello, everyone. Welcome to the Direct Technologies, Davood for Thought podcast. I'm Davood Ghods and I'll be your host today. Later. The way I stay up with the pressing topics of tech and government of today is to tap into the panel of experts. I've had the honor of connecting with over the years. Today we have Paul Lavalley on the podcast.

 

00;00;48;20 - 00;01;23;15

Davood Ghods

Paul is currently the CIO of Yuba County and has over 25 years of progressive experience leading IT. Teams that help organizations effectively use technology to meet and exceed business objectives. He has been able to successfully build i.t groups that work with and support all business functions, resulting in improved metrics and satisfaction with its services, all without sharing more about your background. I want to welcome you to this episode of our podcast and ask you to tell us more about your professional experience, what you have done and what you're currently working on. Welcome.

 

00;01;23;22 - 00;01;57;19

Paul Lavalley

Thank you very much. Good to be here. So I'll give you a brief overview of my background. Not going to go back to my birth and childbearing days, but at least professionally, I started life in the San Francisco Bay Area after going to school and growing up in the Midwest.

 

00;01;57;20 - 00;02;35;04

Paul Lavalley

So I actually started at Chevron in their information technology group after getting an undergrad in computer engineering from Michigan State University. If anyone detects a slight Midwestern accent, it's highly possible, even though I've been out here for a long time. While at Chevron, I realized that I wanted to get more business exposure. Since I had gone to the technical background and thankfully Chevron had one of the best of all corporate perks a tuition reimbursement program.

 

00;02;35;06 - 00;03;14;29

Paul Lavalley

So I took advantage of that and I got my MBA from the UC Berkeley Evening program while I worked during days. So after that, I did brief stints in finance and consulting. Then I did IT leadership positions. So it director positions mostly for several Silicon Valley startups. During that time, I think I worked for six startups, four of which we actually brought to the stock market with initial public offering.

 

00;03;15;01 - 00;03;42;14

Paul Lavalley

After that time in the Bay Area, I actually moved to the Sierra foothills where I live now. And while I would say that, I mean there's at least partially a professional component of that. The real reason was more personal. I've got a boy and two girls and the girls have always loved horses. Horse property in the Bay Area is kind of expensive.

 

00;03;42;17 - 00;04;09;02

Paul Lavalley

So, you know, we could only really afford that in the in the Sierra foothills. Why there? I took a the position I now have is the CEO for Yuba County and it's headquartered in Marysville. And that's actually my first public sector experience.  

 

00;04;09;03 - 00;04;39;16

Davood Ghods

Great. Wow. I should say the least. You have had an interesting background and experience is, if not impressive. Thank you for that all. I think you would agree that adjusting to the pandemic has been challenging for many organizations and now everyone is thinking of what the next major disruption like the pandemic is going to be and how can we be better prepared for it. So resiliency, which is a service that we offer, a direct technology and resilience as a service, is a big topic of conversation these days. What are some examples of resilience you've seen in the past year and what is the one thing organizations should be doing to improve resilience?

 

00;04;39;18 - 00;05;12;10

Paul Lavalley

So I will at least talk about what at the county we've done for resilience. Obviously for the pandemic, we did have remote worker capability, but we had to scale that up a lot. So as well as obviously deploying lots of of mobile devices, laptops, tablets.

 

00;05;12;13 - 00;05;43;19

Paul Lavalley

So we definitely had to scale for remote workers in general. The way we look at and I look at resilience is, you know, service availability, How can you make sure critical services are available? No, no matter what happens and then think about what could potentially happen. So as part of thinking about that, we're always looking to understand and eliminate single points of failure and as well provide fast recovery.

 

00;05;43;19 - 00;06;20;18

Paul Lavalley

If something does fail, how do we get it back online in a hurry for the county? A big issue is water based disasters. Marysville is down river from three major dam systems, so the biggest being the Oroville Dam and Marysville has two different levee systems that protect low lying areas. And one of the ways the county kind of started with resiliency is putting a data center in each of those levee systems.

 

00;06;20;20 - 00;06;48;22

Paul Lavalley

So just in case one of them failed, we could recover to the other. The Oroville Spillway, For those of you that were in California or at least the Sacramento area in 2017, you may recall a Orrville Dam spillway incident where the spillway almost failed and potentially would have taken out the dam. So when that happened, we realized that both levee systems could potentially fail.

 

00;06;48;24 - 00;07;21;02

Paul Lavalley

So what we did was started looking more to to the cloud. Since we're a government agency, we have to kind of implement, you know, government cloud more security. We use both. We're Microsoft Shop, so we use both Microsoft 365 as well as we use Azure for infrastructure. And especially on the infrastructure side, the key direction for us and using the cloud is as a spot for for data protection.

 

00;07;21;02 - 00;07;50;24

Paul Lavalley

So kind of to move backups there as well as potentially to be able to use it as a D-R site if our primary data centers were lost and as part of that effort. You know, we've tried a couple of different combination ones of technologies and systems, you know, for data protection to make that as easy as possible, to backup our on prem and cloud instances to both on prem and cloud for for recovery.

 

00;07;50;27 - 00;08;16;28

Paul Lavalley

And we migrated to a system by a company called Rubrik a few years ago. And it has helped us both kind of minimize that effort to do that as well as maximize the flexibility to allow us to recover either on prem or to the cloud. Very good. Well, thank you for providing some background information about your county and your environ.

 

00;08;17;00 - 00;08;49;07

Davood Ghods

I read recently and if you would like to talk about the incident that happened with the ransomware during not too long ago. Can you please tell us what happened? How did you recover and what would have potentially prevented this attack?  

 

00;08;49;09 - 00;09;16;27

Paul Lavalley

Sure. So first of all, what happened and this was in February of this year, the county was hit by a ransomware attack from technology called Dapo Palmer. And the brunt of the attack happened, as many of them do in the middle of the night. And Yuba County is a relatively small county. We don't have 24 by seven network of security operations. So we but we do have an on call person because we obviously we do have 24 seven operations for sheriff and jail and juvie hall and some other county operated agents.

 

00;09;16;29 - 00;09;48;19

Paul Lavalley

So we got a call at two in the morning. And after a little bit discovery, we realized what was going on. I mentioned kind of the rubric data protection system. Thankfully, we had it. Well, we didn't have everything. We did have some mistake in missed servers, but for everything we did have backed up on rubric. We were fully able to recover all data and we were eventually able to able to restore services.

 

00;09;48;21 - 00;10;14;25

Paul Lavalley

And then, you know, for your listeners that are interested and I did speak at a user conference for Rubrik earlier this year, and there there was also a tech wire article that kind of summarized my presentation. So that's definitely available for people to get more information. And then actually that's about, you know, how did we recover? And let me also address because you also asked about how would we have potentially prevented the attack.

 

00;10;14;27 - 00;10;43;09

Paul Lavalley

Right. And that's almost a bigger question. And I before we talk about what we could have potentially done, we have to talk a little bit more about the details of what happened. And obviously, a lot of this we discovered through the cyber forensics after the attack. But based upon that, the general steps of what happened and in this for from talking to other security professionals, this happens a lot.

 

00;10;43;10 - 00;11;09;10

Paul Lavalley

So this is kind of the primary attack method for a lot of ransomware attack. First thing and end user computer was compromised. And unfortunately, with cyber forensics, we weren't able to determine the exact vector or a compromise. So we don't know the exact file or how it got in. You know, there are some things we know, but but we don't unfortunately know that detail.

 

00;11;09;12 - 00;11;46;20

Paul Lavalley

Once that end user computer was compromised, it was used as a probe to look across our network for vulnerabilities. The vulnerabilities attackers do seem to look for are especially any type of administrative system that doesn't support multifactor or two factor authentication or is obviously misconfigured in some way. So what they eventually found for us was a Active Directory or not, and I may use the term add as an acronym for Active Directory kind of moving forward.

 

00;11;46;22 - 00;12;21;27

Paul Lavalley

And that also from talking to other cyber security experts does seem to be a well-known target for ransomware attackers. So based upon, you know, that kind of, you know, happening and by the way, I should mention that ad was then used to propagate some of the actual ransomware distribution on devices. But knowing that end user device was attacked and then AD was the primary administrative system that was compromised, know how could we have prevented it?

 

00;12;22;00 - 00;12;46;15

Paul Lavalley

So one of the things we've definitely been working on is improving endpoint protection as well as kind of user security awareness. By the way, we did have both advanced email filtering as well as kind of web filtering in place. So we don't believe it came the initial attack came through a phishing attack, which a lot do seem to.

 

00;12;46;17 - 00;13;27;10

Paul Lavalley

But definitely any investment in endpoint detection and response upgrade for all and end devices, money well spent from a security perspective as well is proactive and responsive vulnerability analysis, automated security updates, anything else that can kind of protect endpoints and users on endpoints are all kind of would be valuable security measures to take. The next is ad hardening and quick aside to kind of indicate the value of multifactor or two factor authentication.

 

00;13;27;13 - 00;14;22;10

Paul Lavalley

I mentioned that we had both Azure infrastructure instances and Microsoft 365 instances that we used for for Cloud when we implemented those services. But we required multi-factor authentication for all admin account access during the attack of our on prem infrastructure. None of our cloud infrastructure was touched. So due to the fact that we had multifactor authentication, we know that our cloud was much more safe than our on prem and why we've looked at ways to and by the way, at least Microsoft Cloud instances use what they call Azure Active Directory or Azure AD for the cloud equivalent of evade.

 

00;14;22;13 - 00;14;55;09

Paul Lavalley

But unfortunately, you can't just flip on prem resources over to Azure. Maybe you still have to use on prem add for a bit. So for those organizations that are still having to use on prem and there are some things that you can do to be more secure obviously with kind of a least privilege and isolation mindset. You know, you can limit all accounts for domain and admin access.

 

00;14;55;11 - 00;15;28;19

Paul Lavalley

So only have the accounts that you absolutely need, have domain or enterprise admin access as well. You want to limit wherever possible the use of those accounts to secure boxes, whether it's, you know, virtual box or physical box. And then you may even want to consider on those boxes securing the access to the box with third party MFA or something else too, to make them even more secure.

 

00;15;28;22 - 00;16;02;26

Paul Lavalley

And then, you know, thirdly, obviously all password protections that you can implement anything for complexity, anything for aging, all of the other kind of missed requirements, you know, that you should fully implement. So and all of that hopefully will allow organizations to at least harden their their use of add on premises until they can they can migrate to a system that fully supports going to multi-factor authentication, at least for for admins.

 

00;16;02;28 - 00;16;35;14

Paul Lavalley

And then the last thing that we're looking at is since many of these attacks are targeted, when it's people are not there to respond, a lot of them are targeted to either happen after hours and weekends, holidays, you know, times when live IP people may not be there to actively respond. You should look at whatever you can do for network level intrusion detection and prevention.

 

00;16;35;16 - 00;17;04;18

Paul Lavalley

And again, for organizations like us that don't have 24 by seven security operations, you know, look to automated systems or look to services that that may enable that type of 24 by seven network coverage. It may have been able to identify and then shunt the connections that were used as part of the attack and may have been able to prevent everything.

 

00;17;04;25 - 00;17;28;29

Davood Ghods

Thank you so much, Paul, for sharing all of that. For our listeners, for our audience who may benefit from that. Definitely, digital safety is key these days to make sure that you don't come across a situation like you had to experience. And I'm so glad that you were not one of those entities that had to pay the ransom.

 

00;17;28;29 - 00;18;10;09

Davood Ghods

Where we have a digital safety assessment service that we provide that allows for and user awareness, we provide endpoint protection, privilege, access management and security and best practices. And we detect and protect. We have a service that offers that 24 by seven service that you mentioned. So that's great. Good luck to you and your team. I want to get away a little bit from the technical aspects and ask you a question about motivation, direct technology.

 

00;18;10;09 - 00;18;37;20

Davood Ghods

We always talk about how we are going to get a project done, what we also ask why we are doing what we are doing. What is your why? In other words, what motivates you in your work?  

00;18;37;20 - 00;19;16;08

Paul Lavalley

Several things that I think motivate me and most people, you know, personally, kind of starting with the technical background and kind of, you know, thinking a little bit like an engineer. I like solving problems, you know, I think solving problems for most organizations and enterprises doesn't just involve technology. It also involves kind of people and business processes. So and, you know, from from my experience, you have to kind of work and improve all three ultimately in order to make organized actions, you know, more effective and efficient. I think for for myself as well as I see a lot of others and kind of assume all my employees and other things have it and kind of often ask as part of the interview process.

 

00;19;16;10 - 00;19;42;12

Paul Lavalley

You know, I think there's also a most people have a need for fulfilling it through important work and hopefully also a need to kind of help people. And then the last thing I kind of think about in general as part of, you know, in any role I've had or job I've had is trying to improve things and make things better than how I found them.

 

00;19;42;14 - 00;20;12;15

Davood Ghods

Excellent. Those have been my motivators also being fulfilled in the job and trying to leave them better than the way I found them. Exactly. In any team, you have to encourage your team members to be innovative. So what inspires innovation on your team? How do you make your team to be more innovative? What are the kinds of things you would do to make sure that they are inspired about being more innovative?

 

00;20;12;17 - 00;20;37;23

Paul Lavalley

So I think in I've worked for small county government, I've worked for lots of startups, I've also worked for Big Oil. So that's a little bit different in terms of resources. But for a lot of the smaller organizations, resources have always been an issue. So definitely I would say that from my background, necessity is always the mother of innovation and invention.

 

00;20;37;25 - 00;21;26;19

Paul Lavalley

And so as part of that, I've always tried to provide context of that, that need, you know, what do we need? Why do we need it? Why is it important for the organization so that the team at least knows what the objectives are, that innovation will ultimately solve other things? I think as part of, you know, as a manager and director, you know, making sure people understand the need and then using an open mind when people come back with possible solutions and making sure as part of that process and whether it's brainstorming or whatever, you know, you get feedback from multiple people when you're looking at solutions.

 

00;21;26;22 - 00;21;57;15

Paul Lavalley

And then you have to realize, you know, a great idea can come from anywhere. The other thing that's important is not just kind of when you get to the to the innovation or invention, but you also have to make sure to recognize and encourage the team. And I would say it's important in two areas, you one, it's important, you know, whoever kind of came up with that idea, it's important to recognize them, reward them to the extent that you can.

 

00;21;57;18 - 00;22;33;01

Paul Lavalley

And then it's also important to recognize and reward those that ultimately helped implement that idea. Yes, all great points. But in allowing inspiring innovation, I always also think that you have to give them the room to fail. You got to let them try things and give them. If it's not a real environment, then a test environment or a sandbox that they can try things and be able to fail so they can succeed.

 

00;22;33;03 - 00;22;54;12

Paul Lavalley

I would just add that as well, and I fully agree. And when I was talking about that open mind while considering solutions, I think it's also important as part of that to, you know, some solutions may may not be the most effective, but you still want them because you may not know until you try them out, right, in a test environment.

 

00;22;54;12 - 00;23;25;07

Davood Ghods

So I totally agree with what you said. Exactly. A couple of personal questions. Kind of we always try to get something personal out of our guests. So my next question is about your background or interests. What is something that would surprise people about your background or interests that they don't might not know about before kind of moving in to I.T. leadership, whether in private sector or public sector?

 

00;23;25;09 - 00;23;58;26

Paul Lavalley

I was an athlete, so I actually was a football player, wrestler, track athlete. And and at some point in in my past, I was planning on an NFL career or some other professional sports career. But before signing with one of several colleges that were recruiting me, I was injured. So I, you know, blew a knee. So most I guess I'm sure some i.t people have athletic backgrounds too, but some were.

 

00;23;58;29 - 00;24;26;06

Paul Lavalley

Yeah, thankfully. And you know this because you've been trying to schedule me for a while and I've been putting it off and putting it off of that knee that I injured. I recently had replaced surgically and the recovery from that, I can just say, is more daunting than I anticipated. But it's not. It's now. Now I'm almost back to normal.

 

00;24;26;08 - 00;24;52;01

Davood Ghods

Very good. Very good. Well, I'm glad you're back to normal. And yes, I do know some not too many, but some other i.t colleagues who had athletic background and football players actually. So good to have another one. Thank you for that answer. And my last question, paul, where can people find you and keep tabs on what you're working on?

 

00;24;52;04 - 00;25;26;16

Davood Ghods

Is there a website? How can people support your work? I do kind of update, you know, basic career information on LinkedIn, so that's the best place to look. Excellent. Okay. Well, thank you so much for joining us today, Paul. Thank you to all the listeners out there for joining us as well. We will see you in the next episode of Dog Food for Thought, where we will shed more light on the human side of tech.

More from Podcasts

Launch Consulting Logo
Locations