libmboard hangs in random iterations

[zauster]

This is the serious error in FLAME-HPC/xparser#14, also an issue of libmboard (and not xparser).

Using the attached minimal example (it is really minimal: a Sender agent sends a message to a Receiver agent), build and run like this:

xparser -p -f bl_model.xml     ## with or without the -f flag
make
mpirun -np 2 ./main 2000 its/0.xml -r

hangs/stalls/deadlocks at seemingly random iterations with full CPU usage! Sometimes the simulation finishes, sometimes not. If I use at least four processes mpirun -np 4 --oversubscribe ./main 2000 its/0.xml -r, then the stalls occur in every simulation run (again at random iterations).

I tracked the error to the MB_SetAccessMode function and in there to (I think) the MPI_Allgather call here.

I can replicate this behaviour with both OpenMPI 3.0.1 and MPICH 3.2.1 (both using the MPI 3.0 Standard).

I haven't found a workaround for this problem, so I am unable to run any simulations at the moment. So this is a quite serious error (at least for me). Any help on this is greatly appreciated!

[zauster]

Using OpenMPI 2.1 shows the same error...

[svdhoog]

Looking at the minimal example xml file, what strikes me is that the agents do not have an ID variable. Instead, the Sender type has Sender_ID and the Receiver agent type has Receiver_ID. Could you try with simply id in lower case (and uppercase ID in the C code)?

[zauster]

Renaming does not change it, unfortunately. The exact same problem occurs.
I just chose other names to be more explicit...

[zauster]

For completeness, the updated minimal example.

[mondus]

@svdhoog can you confirm if you have had any similar problems with your large MPI models? What is your current MPI config?

[svdhoog]

@mondus Well in fact our model still fits on one CPU, so we are doing distributed rather than parallel computations. Parameter sweeps etc. So I haven't bumped into this problem. Aren't there a bunch of unittests in Libmboard? At the configure stage, I recall one can set a flag --disable-tests, so the standard config must install those unit tests? The Readme mentions:

./tests/run_test_parallel_utils: Test utility code used only by the parallel modules
./tests/run_test_parallel : Test parallel libmboard API

Could be worth a try to check if all tests pass?

[zauster]

@svdhoog @mondus true, that's worth a try. Unfortunately now, I get a segfault when I want to carry out the tests. At first glance it looks like the segfault in issue #5.

The gdb backtrace looks very similar:

(gdb) bt
#0  0x00007ffff7bccaec in CU_register_nsuites () from /usr/lib/libcunit.so.1
#1  0x0000555555555800 in testsuite_mb_serial () at setup_suite_mb_serial.c:97
#2  0x0000555555555738 in main (argc=1, argv=0x7fffffffe038) at run_test_serial.c:48
(gdb) 

[Bundenth]

@zauster something odd here, running the tests I only encounter the segfault error when compiling libmboard linked to CUnit-2.1.3, but if I use CUnit-2.1.2 I do not see any errors and the tests run successfully.

However I do experience the hangs you mentioned in your minimal example, when running mpirun with 4 processes...

[Bundenth]

Not sure if what I am seeing is the expected behaviour, but every time one simulation hangs, going to the profiler I can see all processes consuming CPU but one, which remains at 0% CPU utilisation. If for whatever reason one of the processes is hanging without much to do and the library calls MPI_AllGather, then you would see a halt as one process would be stalling all the others.

I am not sure how @zauster came to the conclusion that the problem lies with an MPI_Allgather call, but I will see if I can trace the problem.

[zauster]

Downgrading CUnit to 2.1.2 does indeed solve the test issue, but not the random stopping. That still persists

[Bundenth]

After some digging I can confirm the hanging in MPI_Allgather that @zauster suggested is not the cause of the problems, but rather a symptom. All processes need to reach an MPI_Allgather to complete communication, but one of the processes seems to hang in another execution point.

By doing some debugging on the template file used by xparser (main.tmpl) I have narrowed down the area where one of the processes hangs to the code that handles "Clear message boards that have finished being used and sync complete if doing late sync complete", between lines 877 and 940 in main.tmpl file. The process that hangs (0% CPU usage) stays in this part of the code at iteration X, whilst the other processes continue to iteration X+1; once the processes reach the MPI_Allgather call, they wait for the hanging process, which never enters the MPI call.

I am not very familiar with Flame in general, so it is difficult for me to figure out what is going on. Maybe someone else can look into this?

[zauster]

So comparing the template code and the code from the example, the only left culprit is MB_SyncComplete call then?
Aside: Isn't it also false that the whole rc = MB_Clear(...) block from the template is not included in the written main.c? From my interpetation of the FLAME template code that block should also be included in the main file...

[Bundenth]

OK some more interesting results. After seeing that processes where hanging in different iteration loops, I thought it would be interesting to see if the hanging still appears when we force all processes to finish the iteration before moving on (with the use of MPI_Barrier())... and it indeed stops the hanging! Basically inserting barriers before and after any synchronisation call in the main.c generated file stops the issue (although because I am no expert in FLAME, I am unsure if this is affecting the simulation results).

One reason this may be happening is that MPI messages at different parts of the code are being confused. To illustrate my point here is a simple example: an application sends and receives messages in two phases:

SendAndReceiveMessagesType1();
...
SendAndReceiveMessagesType2();

Now because there is no explicit synchronisation barrier between both calls, some processes may enter SendAndReceiveMessagesType2() before some have finished SendAndReceiveMessagesType1(). Hence, one process may be wrongly receiving a type1 message in the type2 function, with unexpected results.

Normally this problem can be solved easily using flags within MPI calls, and a good candidate for a flag can be the iteration number, or something unique so messages of different types are not mixed.

I do not know if this is the case in FLAME, but it is worth looking into. @zauster at least you have a workaround for your experiments (adding MPI_Barriers as in the attachment).

main.c.zip

Aside: Isn't it also false that the whole rc = MB_Clear(...) block from the template is not included in the written main.c? From my interpetation of the FLAME template code that block should also be included in the main file...

Not sure what you mean, when I xparse your project I do see the call to MB_Clear() included.