Is it possible to lock some data in CPU cache?

Question

I have a problem.... I'm writing a data into array in the while-loop. And the point is that I'm doing it really frequently. It seems to be that this writing is now a bottle-neck in the code. So as i presume it's caused by the writing to memory. This array is not really large (smth like 300 elements). The question is it possible to do it in that way: to store it in the cache and update in the memory only after the while-loop is finished?

[edit - copied from an answer added by Alex]

double* array1  = new double[1000000]; // this array has elements  
unsigned long* array2  = unsigned long[300];
double varX,t,sum=0;
int iter=0,i=0;
while(i<=max_steps)
{
   varX+=difX;
   nm0 =  int(varX);
   if(nm1!=nm0)
   {
        array2[iter] = nm0;  // if you comment this string application works more then 2 times faster :)
        nm1=nm0;
        t = array1[nm0]; // if you comment this string , there is almost no change in time 
        ++iter;
   }
   sum+=t;
   ++i;
}

Answer 1

I doubt that this is possible, at least on a high-level multitasking operating system. You can't guarantee that your process won't be pre-empted, and lose the CPU. If your process then owns the cache, other processes can't use it, which would make their exeucution very slow, and complicate things a great deal. You really don't want to run a modern several-GHz processor without cache, just because one application has locked all others out of it.

Answer 2

Not intentionally, no. Among other things, you have no idea how big the cache is, so you have no idea of what's going to fit. Furthermore, if the app were allowed to lock off part of the cache, the effects on the OS might be devastating to overall system performance. This falls squarely onto my list of "you can't do it because you shouldn't do it. Ever."

What you can do is to improve your locality of reference - try to arrange the loop such that you don't access the elements more than once, and try to access them in order in memory.

Without more clues about your application, I don't think more specific advice can be given.

Answer 3

Unless your code does something completely different in between writing to the array, then most of the array will probably be kept in the cache.

Unfortunately there isn't anything you can do to affect what is in the cache, apart from rewriting your algorithm with the cache in mind. Try to use as little memory as possible in between writing to the memory: don't use lot's of variables, don't call many other functions, and try to write to the same region of the array consecutively.

Answer 4

Even if you could, it's a bad idea.

Modern desktop computers use multiple-core CPUs. Intel's chips are the most common chips in Desktop machines... but the Core and Core 2 processors don't share an on-die cache.

That is, didn't share a cache until the Core 2 i7 chips were released, which share an on-die 8MB L3 cache.

So, if you were able to lock data in the cache on the computer I'm typing this from, you can't even guarantee this process will be scheduled on the same core, so that cache lock may be totally useless.

Answer 5

If your writes are slow, make sure that no other CPU core is writing in the same memory area at the same time.

Answer 6

It might be possible to use some assembly code, or as onebyone pointed out, assembly intrinsics, to prefetch lines of memory into the cache, but that would cost a lot of time to tinker with it.

Just for trial, try to read in all the data (in a manner that the compiler won't optimize away), and then do the write. See if that helps.

Answer 7

The CPU does not usually offer fine-grained cache control, you're not allowed to choose what is evicted or to pin things in cache. You do have a few cache operations on some CPUs. Just as a bit of info on what you can do: Here's some interesting cache related instructions on newer x86{-64} CPUs (Doing things like this makes portability hell, but I figured you may be curious)

Software Data Prefecth

The non-temporal instruction is prefetchnta, which fetches the data into the second-level cache, minimizing cache pollution.

The temporal instructions are as follows:

* prefetcht0 – fetches the data into all cache levels, that is, to the

second-level cache for the Pentium® 4 processor.

* prefetcht1 – Identical to prefetcht0

* prefetcht2 – Identical to prefetcht0

Additionally there are a set of instructions for accessing data in memory but explicitly tell the processor to not insert the data into the cache. These are called non-temporal instructions. An example of one is here: MOVNTI.

You could use the non temporal instructions for every piece of data you DON'T want in cache, in the hopes that the rest will always stay in cache. I don't know if this would actually improve performance as there are subtle behaviors to be aware of when it comes to the cache. Also it sounds like it'd be relatively painful to do.

Answer 8

When you have a performance problem, don't assume anything, measure first. For example, comment out the writes, and see if the performance is any different.

If you are writing to an array of structures, use a structure pointer to cache the address of the structure so you are not doing the array multiply each time you do an access. Make sure you are using the native word length for the array indexer variable for maximum optimisation.

Answer 9

As other people have said, you can't control this directly, but changing your code may indirectly enable better caching. If you're running on linux and want to get more insight into what's happening with the CPU cache when your program runs, you can use the Cachegrind tool, part of the Valgrind suite. It's a simulation of a processor, so it's not totally realistic, but it gives you information that is hard to get any other way.

Answer 10

I have a problem.... I'm writing a data into array in the while-loop. And the point is that I'm doing it really frequently. It seems to be that this writing is now a bottle-neck in the code. So as i presume it's caused by the writing to memory. This array is not really large (smth like 300 elements). The question is it possible to do it in that way: to store it in the cache and update in the memory only after the while-loop is finished?

You don't need to. The only reason why it might get pushed out of the cache is if some other data is deemed more urgent to put in the cache.

Apart from this, an array of 300 elements should fit in the cache with no trouble (assuming the element size isn't too crazy), so most likely, your data is already in cach.

In any case, the most effective solution is probably to tweak your code. Use lots of temporaries (to indicate to the compiler that the memory address isn't important), rather than writing/reading into the array constantly. Reorder your code so loads are performed once, at the start of the loop, and break up dependency chains as much as possible.

Manually unrolling the loop gives you more flexibility to achieve these things.

And finally, two obvious tools you should use rather than guessing about the cache behavior:

• A profiler, and cachegrind if available. A good profiler can tell you a lot of statistics on cache misses, and cachegrind give you a lot of information too.
• Us here at StackOverflow. If you post your loop code and ask how its performance can be improved, I'm sure a lot of us will find it a fun challenge.

But as others have mentioned, when working with performance, don't guess. You need hard data and measurements, not hunches and gut feelings.

Answer 11

For the first I'd like to thank all of you for answers. Indeed, it was a little dumb not to place a code. So i decided to do it now.

double* array1  = new double[1000000]; // this array has elements  
unsigned long* array2  = unsigned long[300];
double varX,t,sum=0;
int iter=0,i=0;
while(i<=max_steps)
{
   varX+=difX;
   nm0 =  int(varX);
   if(nm1!=nm0)
   {
        array2[iter] = nm0;  // if you comment this string application works more then 2 times faster :)
        nm1=nm0;
        t = array1[nm0]; // if you comment this string , there is almost no change in time 
        ++iter;
   }
   sum+=t;
   ++i;
}

So that was it. It would be nice if someone will have any ideas. Thank you very much again.

Sincerely Alex

Answer 12

In this case, array2 will be quite "hot" and stay in cache for that reason alone. The trick is keeping array1 out of cache (!). You're reading it only once, so there is no point in caching it. The SSE instruction for that is MOVNTPD, intrinsic void_mm_stream_pd(double *destination, __m128i source)