Vector Usage in MPI(C++)

Question

I am new to MPI programming,stiil learning , i was successful till creating the Derived data-types by defining the structures . Now i want to include Vector in my structure and want to send the data across the Process. for ex:

struct Structure{

//Constructor 
Structure(): X(nodes),mass(nodes),ac(nodes) {

//code to calculate the mass and accelerations
}
//Destructor
Structure() {}

//Variables
double radius;
double volume;
vector<double> mass;
vector<double> area;

//and some other variables

//Methods to calculate some physical properties

Now using MPI i want to sent the data in the structure across the processes. Is it possible for me to create the MPI_type_struct vectors included and send the data?

I tried reading through forums, but i am not able to get the clear picture from the responses given there. Hope i would be able to get a clear idea or approach to send the data

PS: i can send the data individually , but its an overhead of sending the data using may MPI_Send/Recieve if we consider the domain very large(say 10000*10000)

Answer 1

Defining structures in MPI is a pain. I think an easier approach would be to use the fact that STL vectors are guaranteed to have contiguously allocated memory. That is, you can treat them like C arrays by getting a pointer to the first element.

std::vector<float> data;
// ... add elements to your vector ...
MPI_Send(&data.front(), data.size(), MPI_FLOAT, 0, 1, MPI_COMM_WORLD);

Answer 2

I'm certainly not an MPI data structure expert, but I don't think that this can be done. The reason is that basically you have a structure with pointers to the data to be sent. All of the MPI datatype functions assume that the data you want to send is in a contiguous area of memory. If the vectors are of a fixed maximum size, and you could do

double radius;
double volume;
int mass_size;
int area_size;
double mass[MASS_MAXLEN];
double area[AREA_MAXLEN];

and then send only the elements that are filled in.

Alternatively, you could pack the data yourself into one array prior to the send and send the array. You would have to do some profiling to see if this was faster than doing separate sends.

Answer 3

I describe my solution to a similar problem here:

http://stackoverflow.com/questions/2345615/message-passing-arbitrary-object-graphs/2380252#2380252

Remember that you can create custom MPI datatypes from prevoiusly defined custom MPI datatypes. For instance, you can define a Struct that describes the layout and content of a single instance of Structure, then a Vector from these Struct datatypes for the entire vector of objects. Heck, if you have several such vectors, you can create a third layer of abstraction, creating a HIndexed datatype from Vectors of Structs, and then send them all with a single message.

In the post I linked above you will find links to the descriptions of the various custom MPI datatypes, that should help you decide on the proper approach.

Answer 4

I do stuff like you're doing, and I know my processor architecture is homogenous. I avoid lots of byte-swapping and lots of MPI packing and unpacking by using Boost serialization.

Sending:

  ostringstream oss;
  {
    binary_oarchive oa(oss);
    oa << BOOST_SERIALIZATION_NVP(myStruct);
  }

  MPI::COMM_WORLD.Send(oss.str().c_str(),
                       oss.str().size(),
                       MPI::Datatype(MPI_BYTE),
                       MpiLibWrapper::DEST_PROC_RANK,
                       MpiLibWrapper::MY_STRUCT_TAG);

Receiving:

    vector<char> incomingBuffer(MAX_BUFFER_SIZE);

    MPI::COMM_WORLD.Recv(&incomingBuffer[0], incomingBuffer.size(),
                         MPI::Datatype(MPI_BYTE),
                         MpiLibWrapper::SRC_PROC_RANK,
                         MpiLibWrapper::MY_STRUCT_TAG,
                         msgStatus);
    if (MpiLibWrapper::ErrorOccurred(msgStatus.Get_error(),
                                     info.logging)) {
      break;
    }
    incomingBuffer.resize(msgStatus.Get_count(MPI::Datatype(MPI_BYTE)));

    if (incomingBuffer.size() > 0) {
      // Shockingly, this does not result in a memory allocation.
      istringstream iss(string(&incomingBuffer[0], incomingBuffer.size()));

      binary_iarchive ia(iss);

      ia >> BOOST_SERIALIZATION_NVP(myStruct);
    }