KEEPING THE LIGHTS ON

BY TJ TRIOLO

Deep Q-learning detects, combats power grid cybersecurity weaknesses

Electrical grids have seen many technological advancements since they began providing power in the 1880s. With computer-controlled systems common among power grids all over the world, systems can deliver power more efficiently than ever before.

However, as power grids rely more heavily on computerbased systems, there also comes increased vulnerability to cyberattacks. A well-designed cyberattack can bring a city to its knees, shutting down the electricity that keeps modern cities bustling, such as when hackers shut down a part of Ukraine’s power grid that supplied more than 230,000 people in 2015.

To prevent such a destructive cyberattack from taking power grids offline, Mohammadamin Moradi, an electrical engineering doctoral student, used artificial intelligence to analyze the most damaging attacks and best defenses possible. His work was guided by two faculty members in the School of Electrical, Computer and Energy Engineering, part of the Ira A. Fulton Schools of Engineering at Arizona State University: Ying-Cheng Lai, a regents professor of electrical engineering and advisor for Moradi’s doctoral degree, and Yang Weng, an assistant professor of electrical engineering.

The research was funded by the U.S. Department of Energy and the Israeli Ministry of Energy through the Israel-United States Binational Industrial Research and Development Foundation, or BIRD Foundation, to help both countries increase their cybersecurity defenses.

Moradi, Lai and Weng worked with a type of machine learning known as deep-Q reinforcement learning, combined with stochastic game theory, to simulate what cyberattacks could cause the most damage to a power grid and the best countermeasures to keep the grid operating as best it can in the face of such attacks. The team’s research paper, “Defending smart electrical power grids against cyberattacks with deep Q-learning,” was recently published in PRX Energy, a highly selective and new open-access journal of the American Physical Society focusing on modern energy issues.

“Power grid security has a substantial impact on Americans’ lives,” Moradi says. “Last year in Texas, there were power outages and people were freezing in winter. A well-planned, well-informed attacker can cause a lot of disasters, and we as defenders should be ready to act accordingly.”

A series of weather events caused the 2021 Texas power crisis, which left more than 4.5 million homes and businesses without power for more than two weeks. A cyberattack on the power grid has the potential to cause similar devastating effects. Moradi explored deep-Q learning in an effort to help bolster power grid cybersecurity.

A Deep Dive into Deep-Q Learning

Deep-Q learning is a subset of Q-learning, a type of machine learning that analyzes the results of inputs and seeks to maximize the reward for an action. In conventional Q-learning, different inputs from a user are mapped to output values in a table known as the Q-table. However, there are many challenges to creating a Q-table because it requires a large amount of computation as the number of input values increases. This can cause a computer to struggle and malfunction when the number of inputs and outputs reaches a certain size, leading Moradi to explore deep-Q learning.

Deep-Q learning varies from traditional Q-learning because it doesn’t need a Q-table, decreasing the capacity needed for computing processes. Instead, deep-Q learning uses a neural network, which is a type of machine learning model, to estimate outputs and does not require the user to make a manual input for each output.

Moradi also chose deep-Q learning because of another factor important to power grid cybersecurity: It can be used for environments where parameters are unknown, just as the optimal attack and defense strategies aren’t known by a user before running the deep-Q learning simulation.

While deep-Q learning addresses the issue of computing power needed, the algorithm model the system uses to learn also needs to be optimized to ensure the best outcome. This is how Moradi picked the idea of modeling the situation as a stochastic game.

Cyber Wargames

“I had a game theory class last semester, and I thought the attacker-defender relationship could be modeled as a game where each one tries to achieve its objectives,” Moradi says. In a two-person stochastic game, two antagonistic parties play a game with multiple stages seeking to maximize their rewards. In this case, the reward for the attacker is to cause as much damage to the power grid as possible, while the reward for the defender is to use the best strategy to minimize damage to the power grid.

“One game move is like one iteration of attacking and defending; the attacker launches an attack, and the defender figures out a way to respond,” Lai explains. “In general, the defender is not able to protect all lines simultaneously because of limited resources. This highlights the need for Q-learning because the defender should wisely select the set of lines to protect.”

Powering Up Power Grid Cyberdefense

Ultimately, Moradi’s simulation isn’t used to defend against the grid in real-time. Simulating cyberattacks and cyberdefense shows power grid weaknesses and the best ways for a human operator to defend against them to minimize cyberattack damage.

The research team says they are looking to further refine their work with future simulations, taking into account factors such as limited financial and human resources for defending the power grid.

Weng, whose research focuses on power systems, machine learning and cyberphysical systems among other areas, says he anticipates that the research can be used for real-life cybersecurity.

“Being part of this project helped me appreciate the great idea of using this model to understand a physical system theoretically using AI solutions,” he says. “This will ensure that the work will transition to the real world, which is the core reason for this research.”