The Argonne Leadership Computing Facility (ALCF), a DOE Office of Science user facility, is in the midst of an innovative storage infrastructure upgrade aimed at reducing the amount of time users have to spend managing the massive amounts of data produced by its supercomputers.

For computational scientists, data management and I/O (input/output) efforts can include transferring files from the computer to storage and moving or retrieving data for analysis.

“I/O is generally considered overhead because it’s time not spent doing computations,” said ALCF Director of Operations Bill Allcock, who is leading the storage upgrade. “The goal is to have a system that moves the data as fast as possible, and as easily as possible so users can focus on the science.”

The ALCF’s operations team has completed the first phase of the upgrade with the installation of a second system to compliment its primary disk storage system, an IBM General Parallel File System (GPFS) that provides 20 petabytes (PB) of usable space and a maximum transfer speed of 240 gigabytes per second (GB/s). The second GPFS configuration brings an additional 7 PB of storage and 90 GB/s of transfer speed. While ALCF projects are now spread across the two filesystems, users seamlessly access their project data from what looks to be a single project root directory.

A Large-Scale Storage Cache

The next phase of the ALCF’s storage upgrade is where the innovation comes in. As a first step, Allcock and his team have already installed 30 GPFS Storage Servers (GSS) between the compute system and the two storage systems. They are currently working with IBM to customize and test a GPFS software feature called Active File Management (AFM).

In essence, this GSS system will serve as an extremely large and extremely fast cache, offering 13 PB of space and 400 GB/s of transfer speed. The idea is that it will act as a buffer to prevent the compute system from slowing down due to defensive I/O (also known as checkpointing), analysis and visualization efforts, and delays caused by data being written to storage.

“We’re basically developing a storage system that looks like a processor,” Allcock said. “To the best of my knowledge, no other facility is doing anything like this yet.”

All projects will write to the cache, and then the AFM software will automatically copy the data to the project storage systems.

Files will be evicted from the cache based on utilization and retention time, but users will still be able to access those files transparently. And because each project will write to a single namespace, it will be easy for researchers to find and retrieve whatever data they are looking for.

“As far as users are concerned, they don’t need to know whether they are accessing data from the cache or from the storage backend,” Allcock said. “They will have the option to check where the data is located, but because the cache is so huge, odds are they will never need to stage the data back into the cache after it has been evicted.”

If all goes according to plan, the cache-like configuration is expected to be available to ALCF users this fall.

A Look Ahead: File Migration to Tape Storage

In the near future, Allcock would also like to extend this concept to the ALCF’s tape storage system, which uses the High-Performance Storage System (HPSS). This would involve using a software tool called the GPFS/HPSS Interface (GHI) to enable data to be moved easily between the ALCF’s tape and disk storage systems.

Like the GSS system, ALCF policy determines when files are moved from disk storage to tape storage. With GHI in place, users will have easy access to data even after it is deleted from disk storage. All files in tape storage will appear in the GPFS metadata and will automatically be migrated back to disk when a user requests it.

From the user’s perspective, the GHI system would help them to manage their data by removing files that have not been accessed from disk over a predetermined amount of time. It also requires no effort to manage data for completed projects as files are automatically transferred from disk to tape.

“The goal of our storage upgrade is hands-off data management,” he said. “Ideally, once the new system is complete, our users won’t have to manage their disk space quotas, they won’t have to worry about copying data, and they won’t have to worry about putting data into tape storage. The removal of these data management roadblocks will allow our users to focus more of their time on science.”

This work was supported by the DOE Office of Science (Advanced Scientific Computing Research program).