Irregular Out-of-core Distribution

Now we will look at the combination of two previous examples - the case when in an out-of-core problem the data is distributed in irregular way. We will need three files: the first one called ''data_reg.file'' with the original data regularly distributed and two files with irregular distribution for computation.

  MPI_File_open(MPI_COMM_WORLD, "data_reg.file", MPI_MODE_RDWR |
                MPI_MODE_CREATE, MPI_INFO_NULL, &file1_reg);
  MPI_File_open(MPI_COMM_WORLD, "data.file", MPI_MODE_RDWR |
                MPI_MODE_CREATE, MPI_INFO_NULL, &file1);
  MPI_File_open(MPI_COMM_WORLD, "data-out.file", MPI_MODE_RDWR |
                MPI_MODE_CREATE, MPI_INFO_NULL, &file2);

In this example we assume the global file placement mode. It means the files are opened collectively by all computing nodes and all nodes have access to all of the files. In data_reg.file the data is logically distributed in a BLOCK fashion, i.e. each node is responsible for one block of data. To partition the data using the Hilbert's curve method we use the files containing coordinates.

  LIP_Datamap_create( LIP_DATAMAP_BLOCK, l, &l_l, LIP_INDEXTYPE_INT, 1, 0,
      &datamap );

  LIP_create_hilbert_distribution_ooc(datamap, &datamap1, fx, fy, fz,
  l_l, ps, sample, vertex,edge,minedge);
  
  LIP_remap_ooc(datamap,&datamap1,file1_reg,file1,l_l, isec_l_l,MPI_DOUBLE, 
                LIP_CREATE_LOCAL | LIP_MOVE_DATA);
     
  LIP_remap_ooc(datamap,&datamap1,file1_reg,file2,l_l,isec_l_l,MPI_DOUBLE, 
                LIP_MOVE_DATA);
  LIP_Datamap_free(&datamap, 0);

Creation of the irregular mapping in datamap1 proceeds in two steps. In the first step the LIP_create_hilbert_distribution_ooc function assigns data to processors; in the second, LIP_remap_ooc with LIP_CREATE_LOCAL flag finds the actual location on the node for each data element. The LIP_MOVE_DATA flag tells the function to copy data into its new locations in the new file. The remapping function can be called more times to remap more files according to one distribution.
The rest of the computation looks in the same way as for a regular OOC example, using files with remapped data and the newly created datamap1 object.

Created by Katarzyna Zając