The Architecture of a New Storage and Retrieval System for Scientific Studies

The proliferation of large data repositories inspires continuing interest in storage systems and organization. This paper describes basic principles and architecture of a new prototype system for data storage and retrieval, called D*, which is designed to support advanced scientific studies in Data Grid environments. Unlike typical storage systems, the D* system enables a tighter integration of data-storage and data-mining technologies. Through the application of innovative retrieval and clustering techniques for high-dimensional data, D* can support high-performance data access and provide scientific applications useful insights into the data that can facilitate subsequent processes of data preparation and data mining. The system scales well with the increasing dimensionality of scientific data and works well on incremental loads of data.