I've done far more with compilers - multiple - than with actual operating system development, but understanding the operating system and operating system concepts is crucial to making good use of it in applications.
You don't have to understand everything about the internals of the operating system, but understanding the services provided and knowing enough about how it is implemented helps a lot. I'd suggest understanding operating systems is probably the more important - but both are important.
It also depends on where you see yourself working in future. Compilers are a relatively specialized topic - you will most likely use many more than you develop or maintain (but the same is true of operating systems too). Operating systems are, in many ways, even more esoteric than compilers.
Finally, it also depends on how much knowledge you already have. Compilers are generally simpler than operating systems. Not much simpler - and a good optimizer is really tough to write. It used to be that compilers don't have to deal with concurrency whereas operating systems do - that is now an over-simplification since some compilers do need to deal with multi-threading code on multi-core chips and understanding how the program will exploit concurrency (or not) at run-time. But the compiler itself still tends to be substantially a sequential process - unlike an operating system. If, on the other hand, you have a decent knowledge of stuff like memory management, disk allocation, threads, etc, then maybe you'd find the operating system more accessible than parts of the compiler.