Moving to Out-of-core Data

Let's consider the following situation: our data array is big enough to fill all our local memory. We don't want to use the paging mechanism and allocate big memory buffer to read the whole array from disk, since we want to use only these elements, that are referenced by our index array, which is quite small. We will see how to use lip in this case.

First, we suppose our data array is located in a disk file, called "data.file". As a local buffer we allocate the array x_l of the size 2*n_l - identical to that of the index array with additional space for ghost-area.

 x_l    = malloc( (n_l + n_l) * (sizeof *x_l) ); 

 MPI_File_open(MPI_COMM_WORLD, "data.file", MPI_MODE_RDWR, MPI_INFO_NULL, 
               &file);

In the inspector phase we have to create an extra structure, called IOBufmap, to provide mapping between disk and memory data placement. As a parameter we give the size of the local memory buffer. To create the schedule we provide the size of the array located on disk. Functions LIP_Schedule_create and LIP_Localize work in the same way for both in-core and out-of core versions. The new thing is the LIP_OOC_localize function, that performs the next step of index translation. The newly created tperm_l array consists of indices that point to a local memory buffer of size n_l, as declared in the IOBufmap structure, and to ghost-area located after it.

  LIP_Datamap_create( LIP_DATAMAP_BLOCK, l, &l_l, LIP_INDEXTYPE_INT, 1, 0,
    &datamap );
  /* create IO bufmap which maps local data buffer on the data array 
   * on disk */  
  LIP_IOBufmap_create(n_l, &bufm);

  LIP_Schedule_create( l_l, &schedule );

  LIP_Localize( datamap, perm, LIP_INDEXTYPE_INT, perm_l, LIP_INDEXTYPE_INT,
    n_l, 0, &schedule, LIP_INDEXINFO_NULL );
    
  LIP_OOC_localize(perm_l, LIP_INDEXTYPE_INT, tperm_l, LIP_INDEXTYPE_INT, 
  		n_l, 0, x_l, MPI_DOUBLE, &bufm, &schedule);

After this we can start the executor phase. We use our IOBufmap structure to get two MPI_Datatypes needed to read data from disk: filetype to set view on a file and memtype to place data in proper memory places in the read function. As a displacement parameter to MPI_File_set_view function we have to give the position of the data block associated with the computing node, as the file is logically distributed (and may also be physically distributed on filesystems that support such distribution).

   LIP_IOBufmap_get_datatype(bufm,MPI_DOUBLE,&memtype,&filetype,
                             LIP_DATA_NEW);
   MPI_File_set_view(file,l_l*rank*8,MPI_DOUBLE,filetype,"native",info);
   MPI_File_read_at(file,0, x_l, 1, memtype, &status);

   LIP_Gather( x_l, x_l + n_l, MPI_DOUBLE, schedule );

The irregular loops are computed using the tperm_l array.

  for (i = 0 ; i < n_l ; i++)
    y[i] = -x_l[ tperm_l[i] ];
  for (i = 0 ; i < n_l ; i++)
    x_l[ i + n_l ] = 0.0;
  for (i = 0 ; i < n_l ; i++)
     x_l[tperm_l[i] ] += y[i];

  LIP_Scatter( x_l + n_l, x_l, MPI_DOUBLE, schedule, LIP_OP_SUM );

After that, the data on disk has to be updated. Once again we receive the required datatypes from IOBufmap and call I/O routines.

  LIP_IOBufmap_get_datatype(bufm,MPI_DOUBLE,&memtype,&filetype,
                            LIP_DATA_ALL);
  MPI_File_set_view(file,l_l*rank*8,MPI_DOUBLE,filetype,"native",info);
  MPI_File_write_at(file,0, x_l, 1, memtype, &status);

Now the arrays on the disk are updated and the final sums can be computed in the same way as in the in-core version.

Created by Katarzyna Zając