What's the best operating system to study in order to write your own x86 operating system from scratch?
Honestly, you should probably not start with an x86 architecture, or even operating systems but maybe something like an 8-bit starter kit, like a basic Fox11 development kit. In college, I wrote my first (and only) OS in Assembly for an M68HC11 processor (the one in the kit).
If you really want to build your own OS from scratch, you've got a long road ahead of you.
It might be difficult to comprehend the source for an entire OS all at once. The tutorials over at osdev.org have a few "bare bones" code samples to get you started.
For my OS class in college we used the Nachos OS Project and implemented that. I did the C++ version, however I think there is also a Java port of this as well. I remember it being very interesting and learning a great deal, even though it was a lot of work.
I think Minix was created for pretty much that exact purpose.
Have fun!
I read this article a while back. You might find it interesting. This guy wrote MINIX back in the day for the very purpose of teaching OS concepts. So it would probably be a good simple OS to study. http://www.cs.vu.nl/~ast/brown/
However, as Martin and Cory mentioned, it's a big chunk to chew.
There is not much point in studying obsolete OS's which is pretty much all current OS's as they tend to have long lives. Have a look at some fresh ideas (although based on tried and true) like Singularity
It all depends on how you want your Operating System to function, if you want a microkernel you should probably study Minix 3, or if you want a monolithic kernel the current linux kernel is a good place to start from (HINT: look in arch/x86/boot, there is some very interesting code in there). However I personally think that you should read through the Intel and AMD manuals, and then do a bit of reading on the osdev.org forums and wiki. They have plenty of code to study, and are generally helpful towards newbies.
I think best way to read many different operating system sources, definitely osdev barebone tutorials, whitepapers on OS research and documentation on your target hardware.
I personally would recommend looking at l4-ka pistachio kernel, written in pretty darn good C++. There are also multiple smaller projects definitely worth checking out, like jimix or pedigree.
Best to stick around osdev forums and wiki - there is a lot of information there already answered - see http://forum.osdev.org and http://wiki.osdev.org
I just wrote my version of x86 kernel from scratch! (for my OS class project) and that was experience I couldn't probably describe. You can find valuable resources at above link.