views:

154

answers:

3

I'm writing a program that for performance reasons uses shared memory (sockets and pipes as alternatives have been evaluated, and they are not fast enough for my task, generally speaking any IPC method that involves copies is too slow). In the shared memory region I am writing many structs of a fixed size. There is one program responsible for writing the structs into shared memory, and many clients that read from it. However, there is one member of each struct that clients need to write to (a reference count, which they will update atomically). All of the other members should be read only to the clients.

Because clients need to change that one member, they can't map the shared memory region as read only. But they shouldn't be tinkering with the other members either, and since these programs are written in C++, memory corruption is possible. Ideally, it should be as difficult as possible for one client to crash another. I'm only worried about buggy clients, not malicious ones, so imperfect solutions are allowed.

I can try to stop clients from overwriting by declaring the members in the header they use as const, but that won't prevent memory corruption (buffer overflows, bad casts, etc.) from overwriting. I can insert canaries, but then I have to constantly pay the cost of checking them.

Instead of storing the reference count member directly, I could store a pointer to the actual data in a separate mapped write only page, while keeping the structs in read only mapped pages. This will work, the OS will force my application to crash if I try to write to the pointed to data, but indirect storage can be undesirable when trying to write lock free algorithms, because needing to follow another level of indirection can change whether something can be done atomically.

Is there any way to mark smaller areas of memory such that writing them will cause your app to blow up? Some platforms have hardware watchpoints, and maybe I could activate one of those with inline assembly, but I'd be limited to only 4 at a time on 32-bit x86 and each one could only cover part of the struct because they're limited to 4 bytes. It'd also make my program painful to debug ;)

Edit: I found this rather eye popping paper, but unfortunately it requires using ECC memory and a modified Linux kernel.

+5  A: 

I don't think its possible to make a few bits read only like that at the OS level.

One thing that occurred to me just now is that you could put the reference counts in a different page like you suggested. If the structs are a common size, and are all in sequential memory locations you could use pointer arithmetic to locate a reference count from the structures pointer, rather than having a pointer within the structure. This might be better than having a pointer for your use case.

long *refCountersBase;//The start address of the ref counters page
MyStruct *structsBase;//The start address of your structures page

//get address to reference counter
long *getRefCounter(MyStruct *myStruct )
{
    size_t n = myStruct - structsBase;
    long *ref = refCountersBase + n;
    return ref;
}
Fire Lancer
I don't know if that will work for me, but upvoted because that's still a nifty trick ;)
Joseph Garvin
Having the reference count within the object is sometimes called "intrusive" reference counting, and some would consider external counters to be the default. You might look at `boost::shared_pointer` or equivalent C++0x `std::shared_pointer` to see an example of the non-intrusive alternative.
Potatoswatter
It will work. You can address a member of a `MyStruct` because the mathematical relation between the `MyStruct*` and the member is well-defined: a simple constant addition. In this example, the mathematical relation between the `MyStruct` and the `long` is a multiplication and an addition, two constants but still trivial.
MSalters
@MSalters: I meant it might not work because I'm not sure I can rework what I have to use contiguous memory.
Joseph Garvin
The two arrays don't actually need to be contiguous. Basically, they could be sparse arrays, where only some bytes are actual MyStruct objects / counters.
MSalters
Err, but there would still have to be a constant size between them, otherwise the relationship wouldn't be a simple addition anymore. That means any sort of normal memory allocation is out ;p
Joseph Garvin
+1  A: 

You would need to add a signal handler for SIGSEGV which recovers from the exception, but only for certain addresses. A starting point might be http://www.opengroup.org/onlinepubs/009695399/basedefs/signal.h.html and the corresponding documentation for your OS.

Edit: I believe what you want is to perform the write and return if the write address is actually OK, and tail-call the previous exception handler (the pointer you get when you install your exception handler) if you want to propagate the exception. I'm not experienced in these things though.

Potatoswatter
I'm not sure what you mean. Are you suggesting that I map the memory all read only, but then 'allow' writes to the reference count member by not crashing on SIGSEGV's triggered by writes to the addresses of reference count members?
Joseph Garvin
Precisely. Actually, in that case, it might be doable: simply perform the write in the signal handler. I didn't realize you wanted to allow reads… forgive me for not reading your entire question ;v) .
Potatoswatter
Seems to me that the signal handler would incur significant cost above what a simple write to the shared memory would otherwise expend. I guess whether or not that cost is affordable would depend on how often you expect the clients to update the reference counters.
Dan Moulding
@Dan: YES. Also depends a lot on the operating system.
Potatoswatter
+1  A: 

I have never heard of enforcing read-only at less than a page granularity, so you might be out of luck in that direction unless you can put each struct on two pages. If you can afford two pages per struct you can put the ref count on one of the pages and make the other read-only.

You could write an API rather than just use headers. Forcing clients to use the API would remove most corruption issues.

Keeping the data with the reference count rather than on a different page will help with locality of data and so improve cache performance.

You need to consider that a reader may have a problem and fail to properly update its ref count. Also that the writer may fail to complete an update. Coping with these things requires extra checks. You can combine such checks with the API. It may be worth experimenting to measure the performance implications of some kind of integrity checking. It may be fast enough to keep a checksum, something as simple as adler32.

Permaquid