Test MPI_Barrier C++

Question

Dear all,
How can I be sure that MPI_Barrier act correctly? What's the method of test for that?
Thank you

Answer 1

Hi

I think that to be sure that the MPI_Barrier is working correctly you have to write a program which is guaranteed to behave differently for working and non-working barriers.

I don't think that @Neeraj's answer is guaranteed to behave that way. If the barrier is working correctly the processes will all write their first output lines before any writes a second output line. However it is possible that this will happen even in the absence of the barrier (or where the barrier has failed completely if you want to think of it this way). My assertion does not depend on the very short sleep times he suggests (5ms*rank). Even if you suppose that the processes wait (5s*rank) it is possible that the statements would appear in the barrier-imposed order in the absence of the barrier. Unlikely I grant you, but not impossible, especially when you have to consider how the o/s buffers multiple writes to stdout -- you might actually be testing that process not the barrier. Oh you cry even the most inaccurate computer clock will result in process 1 waiting enough less time than process 2 to show the correct working of the barrier. Not if the o/s preemptively grabs processor 1 (on which process 1 is trying to run) for 10s it doesn't.

Dependence on the on-board clocks for synchronisation actually makes the program less deterministic. All the processors have their own clocks, and the hardware doesn't make any guarantees that they all tick at exactly the same rate or with exactly the same tick length.

Nor does that test adequately explore all the failure modes of the barrier. At best it only explores the complete failure; what if the implementation is actually a leaky barrier, so that occasionally a process gets through before the last process has reached the barrier ? Off-by-one errors are incredibly common in programs. Or perhaps the barrier code was written 3 years ago and only has enough memory to record the arrival of, say, 2^12==4096 processes and you've put it on a brand new machine with 2^18 processors; the barrier is more of a weir than a dam.

I haven't thought about this deeply until now, I've never suspected that any of the MPI implementations I've used had faulty barriers, so I don't have a good suggestion about how to thoroughly test a barrier. I'd be inclined to use a parallel debugger and examine the execution of the program through the barrier, but that's not going to provide a guarantee of correct behaviour.

It's an interesting question though.

Regards

Mark