next up previous
Next: Collective I/O Up: Optimization Techniques Previous: Data reusing


Data sieving

Numerous studies of the I/O characteristics of parallel applications have shown that many applications need to access a large number of small, noncontiguous pieces of data from a file [*]. For good I/O performance, however, the size of an I/O request must be large (in the order of megabytes). The I/O performance suffers considerably if applications access data by making many small I/O requests. What is more, to reduce the effect of high I/O latency, it is critical to make as few requests to the file system as possible. Due to the reasons mentioned above, an optimization technique called data sieving was introduced.

Instead of accessing each contiguous portion of the data separately, a single contiguous chunk of data starting from the first requested byte up to the last requested byte is read into a temporary buffer in memory. Additionally, the requested portions of data are extracted and placed in the user's buffer.

A potential problem with this simple algorithm is its memory requirement. The temporary buffer into which data is first read must be as large as the total number of bytes between the first and last byte requested by the user. To avoid this problem, the technique described above is performed in parts, reading only as much data at a time as defined by a special parameter.

The advantage of data sieving is that data is always accessed in large chunks, although at the cost of reading more data than needed. For many common access patterns, the holes between useful data are not unduly large, and the advantage of accessing large chunks far outweighs the cost of reading extra data. In some access patterns, however, the holes could be so large that the cost of reading the extra data outweighs the cost of handling such cases as well. The algorithm can analyze the user's request and decide whether to perform data sieving or access each contiguous data segment separately. Data sieving can similarly be used for writing data.

The library is based on the MPI-IO standard and tested on its portable implementation called ROMIO which implements the technique described above by means of MPI derived data types.

next up previous
Next: Collective I/O Up: Optimization Techniques Previous: Data reusing
Created by Katarzyna Zając