Using Irregular Distribution

Until now we have been using block distribution which meant we divided the arrays into blocks of equal size without taking into account any potential relationships between the data. In many cases, however, we can expect such relationships to exist and we can use them to distribute data in a better way. The case when each array element can be located in any of the nodes is called irregular distribution. A procedure called the partitioner creates such a distribution by using additional information about the data provided by the user. When computation is performed on data located in some 2D or 3D area and we know that the loop represents the computing of some local interaction between geometrically non-distant points, then we can use the coordinate based partitioner. It will distribute data in such a way that it will minimize communication during the executor phase, since most of the data will be available locally.
The next example shows how to use lip to deal with irregular distribution. We take the same loop as before, but we regard all data elements as points in a 2D area. The coordinates of these points are stored in arrays xx[] and yy[]. To do the remapping, we add the following lines:

  LIP_Datamap_create( LIP_DATAMAP_BLOCK, l, &l_loc, LIP_INDEXTYPE_INT, 1, 0,
    &blockdata );
  
  LIP_create_hilbert_distribution_in_core(blockdata,&irrdata,
  xx,yy,NULL,vertex,edge,minedge,sample);
  
  LIP_Schedule_create( l_l, &sch_rem );
    
  LIP_remap_in_core(blockdata,irrdata,&sch_rem);
  LIP_Datamap_free(&blockdata,0);      
  LIP_Schedule_commit( &sch_rem );

  LIP_Gather(x,x2,MPI_DOUBLE,sch_rem);

  LIP_Schedule_free(&sch_rem);

The LIP_remap_in_core function creates a communication schedule that should be used in the LIP_Gather function to move data according to new mapping. The rest of the program is the same as the regular in-core version using the new x2[] data array.

Created by Katarzyna Zając