Multi threaded FFTW 3.1.2 on a shared memory computer

Question

I use FFTW 3.1.2 with Fortran to perform real to complex and complex to real FFTs. It works perfectly on one thread.

Unfortunately I have some problems when I use the multi-threaded FFTW on a 32 CPU shared memory computer. I have two plans, one for 9 real to complex FFT and one for 9 complex to real FFT (size of each real field: 512*512). I use Fortran and I compile (using ifort) my code linking to the following libraries:

-lfftw3f_threads -lfftw3f -lm -lguide -lpthread -mp

The program seems to compile correctly and the function sfftw_init_threads returns a non-zero integer value, usually 65527.

However, even though the program runs perfectly, it is slower with 2 or more threads than with one. A top command shows weird CPU load larger than 100% (and much more larger than n_threads*100). An htop command shows that one processor (let's say number 1) is working at a 100% load on the program, while ALL the other processors, including number 1, are working on this very same program, at a 0% load, 0% memory and 0 TIME.

If anybody has any idea of what's going on here... thanks a lot!

Answer 1

This looks like it could be a synchronisation problem. You can get this type of behaviour if all threads except one are locked out e.g. by a semaphore to a library call.

How are you calling the planner? Are all your function calls correctly synchronised? Are you creating the plans in a single thread or on all threads? I assume you've read the notes on thread safety in the FFTW docs... ;)

Answer 2

Thank you for your answer. I am not sur to understand everything. I am a student in Atmospheric Sciences and it is the first time I hear about a semaphore. I've tried to read and learn about that but I still don't understand what I could be doing wrong. Here is the piece of code where I create the plan. I follow this page:

    call sfftw_init_threads(err)
    print*, err

    call sfftw_plan_with_nthreads(nproc)

    call sfftw_plan_many_dft_r2c(plan_RK,2,size,9,DYN_R,
 .  inembed,1,idist,DYN_K,onembed,1,odist,FFTW_PATIENT)

    call sfftw_plan_many_dft_c2r(plan_KR,2,size,9,DYN_K,
 .  onembed,1,odist,DYN_R,inembed,1,idist,FFTW_PATIENT)

And then I use those plans as much as need, i.e. every time step (finite difference pseudo-spectral model) with a sfftw_execute command. DYN_K and DYN_R are in common blocks. Is this the problem?

Would it be a solution to create one plan for each of the 9 transforms?

Thank you for helping a newbie!

Answer 3

Unless your FFTs are pretty large, the automatic multithreading in FFTW is unlikely to be a win speed wise. The synchronization overhead inside the library can dominate the computation being done. You should profile different sizes and see where the break even point is.