Cybersecurity Lessons: Keep Your City Protected

FEATURE Review by Scott Meyer, City Administrator, Cape Girardeau

As I prepared for the Martin Luther King holiday weekend, I looked forward to some time to stay in town, take in some of the MLK Celebration events, and rest. It had been a mild winter thus far, so I was thankful for the good fortune that might keep the city of Cape Girardeau from dipping into our contingency for snow and ice materials, and overtime hours for this fiscal year. No one had a clue just how different and difficult 2020 was going to be, and the upside-down of 2020 started earlier than most.

The first clue that something was wrong was when my city account email stopped receiving emails. The next day our finance director notified me it was more serious, and by the end of that day, we were setting up an Emergency Operations Center (EOC) meeting to update key city officials and staff and set a course of action.

What we knew in the early hours after discovery of the problem was: • Our email system was down. • Our phone system was not affected. • Network files were inaccessible due to encryption. • Files stored on individual computers were not affected • Data on some of the network storage device(s) was erased. • Nothing on the servers was accessible including utility billing, accounting, licensing, permitting, cashiering, GIS and court software, as well as all daily working files for most departments.

Cloud based software and data was still in place, including 911, e-ticketing, body camera recordings, parks and rec software, etc.

A cyberattack was suspected, and we were concerned that a server was taken down and destroyed. We soon found out it was an attack and ransom was demanded. Our data stores were encrypted by RYUK Ransomware, and hackers, through some type of password keylogger, acquired many of our administrators’ passwords and deleted all of the data from some of the city-owned network storage device(s).

Staff moved on several fronts: • Contacted FBI and its cyberattack unit. • Contacted the insurance company for our cyber insurance. • Developed a plan to keep up our business processes and continue serving customers. 1. Set up Gmail accounts that mirrored our city accounts. 2. Published cell phone options to reach city offices. 3. Began to work on utility billing and set up processes to accept money and credit card payments, as well as cut checks to pay vendors.

Fortunately, staff had just gone to an off-site, time-keeping and payroll system that meant our employees would be paid.

The FBI agent gave staff the authority to tell the public that the City had involved them and were taking the attack seriously. The FBI also provided statistics to share with our citizens. • About 200 public agencies are attacked each year, almost one every other day. • Most attackers are never caught.

Attackers are very sophisticated and helpful to the “client,” even to the point of having a “help desk” to get data unencrypted should there be an issue.

This cyberattack was a true emergency, one that required tapping into all redundant and backup systems to keep the City running, and to rely on our city experts. The insurance company took the lead in the City’s response. It was critical to not do anything that would jeopardize our coverage.

At first, daily meetings were held to work through problems and develop communication strategies, both internally and externally. It was important to manage and coordinate all messaging with the insurance company. The early message was that it was an IT issue, but quickly migrated to a cybercrime that could not be discussed due to the FBI investigation, and then to a more vague description of a cyberattack that included a ransom. At the direction of the insurance carrier, the City did not disclose the amount of ransom paid.

The insurance company set up a negotiation team and arranged for the ransom payment. An IT consultant was assigned and arrived quickly to assess and diagnose the system and develop a plan to bring all systems back online. Forensics testing was completed to determine how the breach occurred, how to prevent it from occurring again, and to determine if the City was negligent in any way.

As staff began to set services in place using redundant methods and developing new ways to work and communicate with customers, staff provided grace periods to utility customers and developed alternate processes to conduct city business. • Suspended utility disconnects and late fees. • Extended license renewal periods. • Waived credit card fees and encouraged customers to use this method of payment. • Handwrote accounts payable checks.

By the second week of February, the de-encryption key was received, and the process began to “clean” the servers and restore the backups. This took about five to six weeks. Unfortunately, our digital plan review software and data were lost as the hackers deleted the storage unit, and the backup storage unit that hosted the application. This system is now being rebuilt from the ground up.

Our basic order of server and file restoration was: • Munis Server: including finance, accounting, cashiering/bill pay, utility billing, licensing and permitting. • Email • Server files (daily working files): police, parks and public works • Engineering and GIS

Insurance Consultant Forensics:

The consultant found only a few clicks on phishing emails. The hacker was rather sophisticated — got in our system, snooped — and gained access to server passwords. The hacker not only encrypted files, but also took down servers that caused several longer-term problems. The consultant used a software that quickly found the ransomware. While the City did have malware installed on all systems, the ransomware was undetected. Hackers write virus code faster than malware can write definitions to catch it. Our IT Manager then worked directly with our own consultant to finish rebuilding servers and develop a comprehensive strategy to address the weaknesses: • Discontinued use of Malwarebytes and Symantec as they were ineffective in stopping the attack. • Installed Falcon by Crowdstrike that was approved by our cyber expert for hacker detection. • Installed Cisco Umbrella client to block endpoints from accessing known bad websites that could spread a virus. • Installed internal firewalls to add another layer of server protection.

Choose a partner who can bring balance to your water and wastewater operations.

FINANCIAL STABILITY

Gain more

control with proactive infrastructure maintenance,

long-term planning, greater buying power, efficient operations and innovative solutions.

AllianceWater.com

Contact Bart Downing to learn more! 816-387-3559 • bdowning@alliancewater.com

Required new sophisticated password protocol with multifactor authentication for 1. Administrative privileged access 2. Remote access all users 3. Remote email access all users Purchased additional backup storage to provide redundant backups and additional protection of data. Moved several servers to off-site hosting in a secure data center to handle the ongoing increase in storage needs. Increased training for end users.

Insurance Renewal

Because of the improvements made, and after several meetings and sharing of information, our insurance policy was renewed for next year. This was received as good news from the City’s insurance broker, given the recent volatility in the public entity cyber insurance marketplace. If the City had to seek coverage alternatives with other carriers, the options would not have been favorable from a pricing or coverage standpoint considering the ongoing claim. Lessons Learned • Buy cyberattack insurance. It is a good resource and provides more than financial help, including expertise in navigating the attack response. • Stay involved in the negotiation even though you may not be in it. In the end, the insurance company requested us to sign off on the amount to be paid,

MU Extension faculty are finding new ways to engage with businesses and communities as we navigate the rapidly evolving impacts of COVID-19.

#ShowMeResilience extension.missouri.edu extension.missouri.edu

we agreed to their assessment that the negotiated amount was the best way to fix our issue, and we supported them by paying that amount under the terms of our policy. It was a tightrope to walk, but something similar was needed to protect the City and make the insurance company comfortable. • Bit Coin is a thing. Because of the ease of transactions between pseudonymous addresses and cryptocurrencies, it is especially attractive to criminals who both exploit technological vulnerabilities and prefer to move funds through these transaction networks to avoid detection by law enforcement. • Banking relationships are important (manual check writing, cash flow etc.) • Have backup processes in place for computerized services. • Look into getting key processes off site and/or cloud based.

It took an extended amount of time to get our utility accounts back in order, and utilities have not been disconnected since the time of the attack, also partly due to COVID. We will be faced with approximately 1,500 of 17,000 accounts being disconnected for non-payment.

Costs

Insurance deductible - $25,000

New virus software on every computer and system - $23,000. The new anti-virus software was at a comparable cost as we had experienced prior.

Cloud storage - $180,000 for three years of service.

Re-installation and configuration of plan review software not able to be restored - $22,725

TOTAL = $250,725

Fortunately, the City had an emergency reserve fund used to cover these expenses.

Impact On Budget Preparation

There was still a lot of data that remained inaccessible in March and April that made it very difficult to prepare the budget in the same way we had in years past. We had to design a different way to put our budget together and make projections. Once COVID-19 hit, many of the numbers became irrelevant. In some ways, it was a good thing because we were able to get a jump on looking at ways to address the budget in a new light.

Conclusions • Have cyber insurance. • Be flexible, nimble and pragmatic when looking for ways to provide continuity of service. • Do not jump to a solution – several times the biggest (most expensive) solution proved to be unnecessary, i.e. moving everything to the cloud, or hosted solutions. When we worked through our backups, we were able to restore and get back most everything. Our bank was able to electronically pay vendors and issued us some checks to hand write to get by. • Work with the insurance company from day one. They have a lot to lose (and gain) by how the problem is resolved. • Be the first and best source of information. Tell what you can and tell why you cannot tell more. • See what you can do today to harden your IT systems to a potential cyberattack.

Scott Meyer has served as Cape Girardeau's city manager for 11 years. A lifelong public servant, he was also the director of facilities for Southeast Missouri State University and a district engineer for MODOT. He earned a Bachelor of Science degree in civil engineering from Missouri University of Science and Technology.

We’re Olsson, engineers who understand that where there’s a project,

there’s a purpose. Meet the team, hear the stories, and learn how Joplin, Missouri, fought its way back from an EF-5 tornado at olsson.com.