views:

658

answers:

7

This question is about the same program I previously asked about. To recap, I have a program with a loop structure like this:

for (int i1 = 0; i1 < N; i1++)
  for (int i2 = 0; i2 < N; i2++)
    for (int i3 = 0; i3 < N; i3++)
      for (int i4 = 0; i4 < N; i4++)
        histogram[bin_index(i1, i2, i3, i4)] += 1;

bin_index is a completely deterministic function of its arguments which, for purposes of this question, does not use or change any shared state - in other words, it is manifestly reentrant.

I first wrote this program to use a single thread. Then I converted it to use multiple threads, such that thread n runs all iterations of the outer loop where i1 % nthreads == n. So the function that runs in each thread looks like

for (int i1 = n; i1 < N; i1 += nthreads)
  for (int i2 = 0; i2 < N; i2++)
    for (int i3 = 0; i3 < N; i3++)
      for (int i4 = 0; i4 < N; i4++)
        thread_local_histogram[bin_index(i1, i2, i3, i4)] += 1;

and all the thread_local_histograms are added up in the main thread at the end.

Here's the strange thing: when I run the program with just 1 thread for some particular size of the calculation, it takes about 6 seconds. When I run it with 2 or 3 threads, doing exactly the same calculation, it takes about 9 seconds. Why is that? I would expect that using 2 threads would be faster than 1 thread since I have a dual-core CPU. The program does not use any mutexes or other synchronization primitives so two threads should be able to run in parallel.

For reference: typical output from time (this is on Linux) for one thread:

real    0m5.968s
user    0m5.856s
sys     0m0.064s

and two threads:

real    0m9.128s
user    0m10.129s
sys     0m6.576s

The code is at http://www.ellipsix.net/ext-tmp/distintegral.ccs

P.S. I know there are libraries designed for exactly this kind of thing that probably could have better performance, but that's what my last question was about so I don't need to hear those suggestions again. (Plus I wanted to use pthreads as a learning experience.)

+1  A: 

One possibility is that the time taken to create the threads exceeds the savings gained by using threads. I would think that N is not very large, if the elapsed time is only 6 seconds for a O(n^4) operation.

There's also no guarantee that multiple threads will run on different cores or CPUs. I'm not sure what the default thread affinity is with Linux - it may be that both threads run on a single core which would negate the benefits of a CPU-intensive piece of code such as this.

This article details default thread affinity and how to change your code to ensure threads run on specific cores.

paxdiablo
In the case I posted N was 20, but I was seeing the same behavior for N=60, which takes 5-10 minutes. So I don't think it's a problem of overhead.
David Zaslavsky
You may need to post bin_index. Six seconds for a 20^4 (160,000) loop seems a little excessive so I'm wondering how complicated it is.
paxdiablo
Well, there's another inner loop that runs ~20 times, which I left out because I didn't think it was relevant to the question. But I did post the code, read it if you like. And thanks for the link about thread affinity.
David Zaslavsky
The increase in time spent in kernel mode suggests the all the threads are on the same core and the kernel is swapping between the two - swapping is very slow. If the threads were on different cores the kernel time shouldn't rise much as there is no swapping going on.
Skizz
+2  A: 

You are seeing cache line bouncing. I'm really surprised that you don't get wrong results, due to race conditions on the histogram buckets.

florin
+1  A: 

Even though threads don't access the same elements of the array at the same, the whole array may sit in a few memory pages. When one core/processor writes to that page, it has to invalidate its cache for all other processors.

Avoid having many threads working over the same memory space. Allocate separate data for each thread to work upon, then join them together when the calculation finishes.

Juliano
I forgot to say so in the original question but actually that is the way the program works. (I've edited it in)
David Zaslavsky
+1  A: 

Off the top of my head:

  • Context switches
  • Resource contention
  • CPU contention (if they aren't getting split to multiple CPUs).
  • Cache thrashing
T.E.D.
+8  A: 

To avoid further comments on this: When I wrote my reply, the questioner hasn't posted a link to his source yet, so I could not tailor my reply to his specific issues. I was only answering the general question what "can" cause such an issue, I never said that this will necessarily apply to his case. When he posted a link to his source, I wrote another reply, that is exactly only focusing on his very issue (which is caused by the use of the random() function as I explained in my other reply). However, since the question of this post is still "What can make a program run slower when using more threads?" and not "What makes my very specific application run slower?", I've seen no need to change my rather general reply either (general question -> general response, specific question -> specific response).


1) Cache Poisoning
All threads access the same array, which is a block of memory. Each core has its own cache to speed up memory access. Since they don't just read from the array but also change the content, the content is changed actually in the cache only, not in real memory (at least not immediately). The problem is that the other thread on the other core may have overlapping parts of memory cached. If now core 1 changes the value in the cache, it must tell core 2 that this value has just changed. It does so by invalidating the cache content on core 2 and core 2 needs to re-read the data from memory, which slows processing down. Cache poisoning can only happen on multi-core or multi-CPU machines. If you just have one CPU with one core this is no problem. So to find out if that is your issue or not, just disable one core (most OSes will allow you to do that) and repeat the test. If it is now almost equally fast, that was your problem.

2) Preventing Memory Bursts
Memory is read fastest if read sequentially in bursts, just like when files are read from HD. Addressing a certain point in memory is actually awfully slow (just like the "seek time" on a HD), even if your PC has the best memory on the market. However, once this point has been addressed, sequential reads are fast. The first addressing goes by sending a row index and a column index and always having waiting times in between before the first data can be accessed. Once this data is there, the CPU starts bursting. While the data is still on the way it sends already the request for the next burst. As long as it is keeping up the burst (by always sending "Next line please" requests), the RAM will continue to pump out data as fast as it can (and this is actually quite fast!). Bursting only works if data is read sequentially and only if the memory addresses grow upwards (AFAIK you cannot burst from high to low addresses). If now two threads run at the same time and both keep reading/writing memory, however both from completely different memory addresses, each time thread 2 needs to read/write data, it must interrupt a possible burst of thread 1 and the other way round. This issue gets worse if you have even more threads and this issue is also an issue on a system that has only one single-core CPU.

BTW running more threads than you have cores will never make your process any faster (as you mentioned 3 threads), it will rather slow it down (thread context switches have side effects that reduce processing throughput) - that is unlike you run more threads because some threads are sleeping or blocking on certain events and thus cannot actively process any data. In that case it may make sense to run more threads than you have cores.

Mecki
Cache poisoning only occurs when both cores don't share the same cache.
Stephen Doyle
@Stephen: This is not true! On a Intel Dual Core each core has its own cache and Intel uses a cache coherency protocol to ensure caches are coherent. However, this protocol will not necessarily update values in other caches, it may just invalidate values there.
Mecki
Point 1 is irrelevant because he is using a thread local store. Point 2 is interesting - is it because of some memory architecture I'm unaware of or because of the cache?
configurator
Regarding your BTW - I think it is wrong. Running more threads per cpu can be faster for certain calculations. I don't know why, but I benchmarked a certain calculation (compiling Expression<Func<int,int>>s) and it was faster with 8 threads than with 4. (I have 2 cores)
configurator
@configurator: See http://en.wikipedia.org/wiki/CAS_latency for details. See table, DDR-400 can beat DDR3-1600! Read the sentence below the table (important). Modern memory is only super fast as long as it can operate in bursts (reading memory sequentially and straight upwards)
Mecki
@configurator: Regarding your function being faster with 8 vs 4 threads on a dual core machine, I'd really like to see the source of the function. The source might explain why it is getting faster with more threads.
Mecki
More threads can go faster IF the host also has other unrelated threads/processes running. In a round robin OS scheduler, the more threads you have the more time you get (in total) on the CPU. If you have 2 of 5 threads then you get 40% of CPU time, 3 of 6 gives you 50%.
Patrick
@Patrick: I thought about this as well. However in that case you would only notice a difference if there are many other "working threads" of other processes. If they are all sleeping, they won't take your CPU time. And if there are many other processes active, this is no valid benchmark environment
Mecki
+3  A: 

Everything I said so far in my other reply holds still true on general, as your question was what "can"... however now that I've seen your actual code, my first bet would be that your usage of the random() function slows everything down. Why?

See, random keeps a global variable in memory that stores the last random value calculated there. Each time you call random() (and you are calling it twice within a single function) it reads the value of this global variable, performs a calculation (that is not so fast; random() alone is a slow function) and writes the result back there before returning it. This global variable is not per thread, it is shared among all threads. So what I wrote regarding cache poisoning applies here all the time (even if you avoided it for the array by having separated arrays per thread; this was very clever of you!). This value is constantly invalidated in the cache of either core and must be re-fetched from memory. However if you only have a single thread, nothing like that happens, this variable never leaves cache after it has been initially read, since it's permanently accessed again and again and again.

Further to make things even worse, glibc has a thread-safe version of random() - I just verified that by looking at the source. While this seems to be a good idea in practice, it means that each random() call will cause a mutex to be locked, memory to be accessed, and a mutex to be unlocked. Thus two threads calling random exactly the same moment will cause one thread to be blocked for a couple of CPU cycles. This is implementation specific, though, as AFAIK it is not required that random() is thread safe. Most standard lib functions are not required to be thread-safe, since the C standard is not even aware of the concept of threads in the first place. When they are not calling it the same moment, the mutex will have no influence on speed (as even a single threaded app must lock/unlock the mutex), but then cache poisoning will apply again.

You could pre-build an array with random numbers for every thread, containing as many random number as each thread needs. Create it in the main thread before spawning the threads and add a reference to it to the structure pointer you hand over to every thread. Then get the random numbers from there.

Or just implement your own random number generator if you don't need the "best" random numbers on the planet, that works with per-thread memory for holding its state - that one might be even faster than the system's built-in generator.

If a Linux only solution works for you, you can use random_r. It allows you to pass the state with every call. Just use a unique state object per thread. However this function is a glibc extension, it is most likely not supported by other platforms (neither part of the C standards nor of the POSIX standards AFAIK - this function does not exist on Mac OS X for example, it may neither exist in Solaris or FreeBSD).

Creating an own random number generator is actually not that hard. If you need real random numbers, you shouldn't use random() in the first place. Random only creates pseudo-random numbers (numbers that look random, but are predictable if you know the generator's internal state). Here's the code for one that produces good uint32 random numbers:

static uint32_t getRandom(uint32_t * m_z, uint32_t * m_w)
{
    *m_z = 36969 * (*m_z & 65535) + (*m_z >> 16);
    *m_w = 18000 * (*m_w & 65535) + (*m_w >> 16);
    return (*m_z << 16) + *m_w;
}

It's important to "seed" m_z and m_w in a proper way somehow, otherwise the results are not random at all. The seed value itself should already be random, but here you could use the system random number generator.

uint32_t m_z = random();
uint32_t m_w = random();
uint32_t nextRandom;

for (...) {
    nextRandom = getRandom(&m_z, &m_w);
    // ...
}

This way every thread only needs to call random() twice and then uses your own generator. BTW, if you need double randoms (that are between 0 and 1), the function above can be easily wrapped for that:

static double getRandomDouble(uint32_t * m_z, uint32_t * m_w)
{
    // The magic number below is 1/(2^32 + 2).
    // The result is strictly between 0 and 1.
    return (getRandom(m_z, m_w) + 1) * 2.328306435454494e-10;
}

Try to make this change in your code and let me know how the benchmark results are :-)

Mecki
If your theory about random() is correct, it's not cache poisoning. It's just the synchronization on the global variable, e.g. a critical section. The solution would be to give each thread their own random number generator state - no need to generate a vast quantity of random data in advance.
Daniel Earwicker
I just checked glibc and it does synchronize, however this seems to be rather an implementation detail - it is nowhere written that random() has to be thread-safe, so you may as well not lock anything. Random() is defined in std C and std C is not even aware of thread existence.
Mecki
@Mecki: Seems like a good idea, I hadn't thought of that. @Earwicker: seems like a better idea ;-) as a typical run needs about a billion random numbers. I think implementing my own RNG might be the way to go here (sounds like fun!)... thanks both.
David Zaslavsky
Yay it works!! ;-) I finally got a chance to put this in my program, and using random_r (which is fine for me because it only has to run on one computer) I now get a factor of 1.5-2 faster with two threads; plus the program as a whole is faster, probably because it doesn't use that random() mutex.
David Zaslavsky
New benchmark results: 1.05s with one thread, 0.66s with 2 threads, for N=20. Or for a longer calculation, N=60: 77.9s with 1 thread, 48.5s with 2 threads.
David Zaslavsky
That is a noticeable speed improvement I'd say :-)
Mecki
A: 

David,

Are you sure you run a kernel that supports multiple processors? If only one processor is utilized in your system, spawning additional CPU-intensive threads will slow down your program.

And, are you sure support for threads in your system actually utilizes multiple processors? Does top, for example, show that both cores in your processor utilized when you run your program?

dmityugov
top does show both cores running at 90-100% while the progarm is running.
David Zaslavsky