Rerouting the Data Traffic Jam

By Casey Moffitt

Researchers at Illinois Institute of Technology, with collaborators from the HDF Group, have brought to market Hermes, a new deep memory and storage hierarchy software system designed to reduce bottlenecks in data movement from storage to memory in high-performance computing.

Improvements in microprocessors, data storage, and memory hardware have drastically increased the amount of data that supercomputers can process, but they also have resulted in traffic jams that occur when moving data from storage to memory. High-performance computing researchers will experience more efficient input/ output performance with Hermes.

“Different systems and different technologies all also run different software,” says Anthony Kougkas (Ph.D. CS ’19), Illinois Tech research assistant professor of computer science and lead researcher on the project. “Ideally, the presence of those multiple tiers of storage should be transparent to applications without having to sacrifice I/O performance.”

Kougkas outlined Hermes’s capabilities as part of the software’s virtual launch in March 2021, with presentations from Hermes team collaborators Christopher Hogan, HDF Group software engineer; Kimmy Hu, HDF Group HPC lead engineer; Hariharan Devarajan (Ph.D. CS ’21) from Illinois Tech’s Scalable Computing Software Lab; and Gerd Heber, HDF Group lead engineer. Illinois Tech Distinguished Professor of Computer Science and Ron Hochsprung Endowed Chair Xian-He Sun is the principal investigator for Hermes, which was funded through a $3 million grant from the National Science Foundation.

Hermes organizes and accesses data from a variety of local and remote storage devices and moves it efficiently into memory, where the data can be processed.

“The problem is all of [the storage devices] are independently designed, deployed, and managed, making it hard for the end user to manage the complexity of hierarchical storage,” Kougkas says. “Hermes holistically manages this storage system and brings the benefit of each [storage component] to the end user.”

Hermes software allows users to draw data from various storage hardware, organize it, and send it to memory using a single software package. This allows the user to write less code with a universal software package, rather than using different software for each storage device.

Although Hermes is fully functional and ready for researchers in high-performance computing to use, Kougkas says the open source software is still “in development,” and he hopes that users will continue to offer their feedback to improve the package. ●

ChronoLog: New Storage System for Activity Data

IN ADDITION TO FUNDING support for the Hermes project, Distinguished Professor Xian-He Sun and Research Assistant Professor Anthony Kougkas have received a National Science Foundation grant to advance a new data storage system that will optimize the efficiency of high-performance computing.

High-performance computing increasingly uses activity data, or log data, which captures information as it occurs. The rapid growth of activity data stresses current data-management systems beyond their capability, creating performance backups. In response, the researchers are developing ChronoLog to organize and store activity data.

Activity data grows with time and is time sensitive. For example, a security camera that is constantly recording needs a place

to store that data. ChronoLog can distribute this activity data across multiple storage devices to effectively scale storage capacity. Older log entries are maintained in a capacity storage tier, while more recent data is stored onto a capability storage tier, creating a feature called log data auto-tiering.

ChronoLog also will have the ability to sequence activity data by leveraging a physical time stamp, avoiding expensive techniques such as synchronization, data sorting, or centralized sequencers.

Illinois Tech will lead this project with $2.7 million of the $4 million National Science Foundation grant. Sun and Kougkas will be collaborating with Kyle Chard, research assistant professor of computer science at the University of Chicago, who was awarded $1.3 million.