IBM Builds Super Fast File System

IBM on Thursday announced that it had scored a breakthrough in file system technology that increases the speed of data access by seven times. Researchers were able to attain a 102-gigabyte per second transfer rate on the ASC Purple supercomputer at Lawrence Livermore National Laboratory in a recent test.

The file system was an astonishing 1.6 petabytes in size, the largest ever in the world, and performance was maintained even as 1,000 clients pushed workloads into the file. The project used 104 Power-based eServer p575 nodes and 416 storage controllers, IBM said in a statement.

The development permits a whole new class of applications, says the company. "Computing capability has been growing very fast, but the file system capacity has not kept up," IBM distinguished engineer Dr. Rama Govindaraju told BetaNews in an interview earlier this week.

Called the General Parallel File System (GPFS), the technology allows for high-speed access to files across multiple nodes of a Linux or AIX cluster. The file system could be used in a variety of fields, including engineering design, digital media and entertainment, data mining, financial analysis, seismic data processing and scientific research.

Govindaraju says that the biggest bottleneck to prevent more feature-rich applications were the file system's capacity and size. "More and more, computers are transitioning from numbers and strings-based to media-based analytics," he added.

With the enhanced capacity of GPFS, entirely new applications have been made possible. For example, the system is already seeing use in medical imaging, and is allowing for doctors to search and compare through thousands of images sometimes hundreds of megabytes in size each.

Uses in the medical field could go beyond just imaging, to enable more intelligent medicine, better medicine design, and for use in educational purposes as well.

The system could also have use in homeland security applications. Govindaraju offered a hypothetical situation where cameras at an airport could be connected to a GPFS-enabled computer allowing for pictures to be taken of passengers at multiple angles and compared to databases of known terror suspects.

"These pictures would have scanned and analyzed before the passenger gets to the immigration officer," he said.

IBM will push GPFS on several fronts, including an effort to even promote its use on non-IBM hardware. The source code behind the file system will be released to eligible clients who can develop upon the technology and share their work with others.

However, the impetus behind the development of GPFS is the user's changing computer needs, says Govidaraju. "The kind of data people are operating on is completely different from even ten years back," he said.