I started an open source "sourceforge" project to make a rich text editor about a year and a half ago. I never figured out how to get my code on there, however to develop it I had to research how the "content editable mode" works in IE and Firefox. My research included mostly the firefox website and the microsoft website.
The code that I saw to do this was ugly and not very OO friendly (sorry I'm an object bigot I can't help it) so it took a lot of re-factoring to get into to a form that I could develop from and extend.
Unfortunately, even if you follow the functionality provided by the browser's "content editable mode" you will still end up with an editor that is full of bugs. The reason for that is that "content editable mode" doesn't work consistently with pasting from MS Word (everyone tries this) or creating numbered lists, and the mark up that it generates will be inconsistent and poorly formed.
This is why I now use TinyMCE. TinyMCE is full of design decisions that I personally would have avoided, but they have fixed the majority of the bugs that you will get when trying to make your own editor. It is also full of features that will allow you to customize it to your needs.
I can't recommend anything else because I haven't really tried the alternatives.
While TinyMCE seems to be the best option, it is still a headache to me because pasting Word documents is still unpredictable, the WYSIWYG promise is not really possible in HTML but the customer expects it, and there are many issues that creep up once you start letting users put raw HTML in your database. (especially when your code base has parts last updated in 1993...)