What is it about?
Key–value stores are being widely used as the storage system for large-scale internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems are the dominant storage solution. In this study, we examine the architecture differences and performance characteristics of parallel file systems and key–value stores. We propose using key–value stores to optimize overall Input/Output (I/O) performance, especially for workloads that parallel file systems cannot handle well, such as the cases with intense data synchronization or heavy metadata operations.
Featured Image
Why is it important?
By utilizing Key-value stores we can optimize I/O performance for small accesses and for workloads with heavy metadata operations. Parallel file systems and key-value stores can work together to offer a high-performance storage access layer.
Perspectives
PFSs are the dominant storage solution in HPC systems. Even though PFSs can offer a high-performance parallel data access, they have some deficiencies with specific I/O workloads such as being noncontiguous and unaligned with the stripe boundary access patterns. Heavy metadata operations are another difficult task for PFSs. On the other hand, KVStores are widely used by internet and cloud storage services, but are rarely used by HPC systems. Those storage solutions can offer an easy-to-use APIs for data manipulation and can offer competitive I/O performance. HPC systems can largely benefit from KVStores. Note that we do not propose to entirely replace PFSs with KVStores. Rather we see them as complementary.
Dr. Anthony Kougkas
Illinois Insitute of Technology
Read the Original
This page is a summary of: Rethinking key–value store for parallel I/O optimization, The International Journal of High Performance Computing Applications, December 2016, SAGE Publications,
DOI: 10.1177/1094342016677084.
You can read the full text:
Contributors
The following have contributed to this page
